Proxy Step

From @mackoj on Wed Apr 20 2016 05:14:15 GMT-0400 (EDT)

Hi,

I’m working on a proxy step.
The goal of this step is to be able to analyze the request made by the app during testing.

In order to make this work I will have to build 2 steps one that enable the proxy and one that disable it and extract the log.

The proxy that I will use is Privoxy.

I will need to do a be root to run networksetup in order define a proxy on OSX.
Can I run sudo in a step?

The way that I see the second step is to be able to run multiple regex against the logfile. In order to do that it will be great if the user can add new inputs in the interface, but I don’t know how to make it in step.yml file.

Thanks,


Copied from original issue: https://github.com/bitrise-io/bitrise-contrib/issues/16

From @viktorbenei on Wed Apr 20 2016 05:22:07 GMT-0400 (EDT)

That sounds really interesting, we’d love to see/have a step (pair) like this :wink: Note: we did the same with our Build Cache steps - you have to add two steps, one for downloading and one for updating&uploading the cache - http://devcenter.bitrise.io/docs/using-the-build-cache

This was the first time we did something like this, a strict step pair which can’t work without each other, so we’ll think about providing better support for the use case. For now, I think, the best is to use a Step title with the same prefix (in our case: Build.io Cache:) and write a warning/highlight in the step’s description that it requires a second step to be added. You can also link to a guide / tutorial from the descriptions if you want to.

About sudo: yes, passwordless sudo is enabled on all of our VMs, so you can install anything you want to :slight_smile:

From @mackoj on Wed Apr 20 2016 10:32:28 GMT-0400 (EDT)

I have a file in my repo that I want to copy where the script actually runs. How can I do that?
For testing purpose I cheat by downloading it where I am but I rather prefer to use the one in the repo.

From @mackoj on Wed Apr 20 2016 10:40:45 GMT-0400 (EDT)

Sorry for closing it. It was a mistake

From @godrei on Wed Apr 20 2016 10:53:22 GMT-0400 (EDT)

Hi @mackoj !
What do you want to do exactly, why do you need that?

From @godrei on Wed Apr 20 2016 11:07:46 GMT-0400 (EDT)

By the way when bitrise runs step, it’s source repository is cloned into a temp directory and bitrise calls bash step.sh, so step.sh is the entry point.
Most of the times, we need to refer to other files in the source repository, this case we use THIS_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" to get the directory where step.sh is located. So if your_ruby_script.rb is in the same directory with step.sh you can refer it with $THIS_SCRIPT_DIR /your_ruby_script.rb

From @mackoj on Wed Apr 20 2016 12:08:35 GMT-0400 (EDT)

Thanks @godrei,

I need it because I want to copy a config file for privoxy and it’s in the repo.

From @viktorbenei on Wed Apr 20 2016 12:13:25 GMT-0400 (EDT)

But why do you want to copy it into the Step’s directory? You should use the file from the repository directly, or put it into a $HOME/ or similar dir AFAIK.

You should not depend on the Step’s code path outside of that step - it might or might not be available after the step finishes to run, when another step runs.

From @mackoj on Thu Apr 21 2016 05:48:59 GMT-0400 (EDT)

This is still in a not finished development version of the script.

In order to use Privoxy I have to start it with a config file privoxy_configfile. This file is in the step repo.

I do not depend on this file in the long term I just need it to start the proxy.

#!/bin/bash

# defining the same proxy setting as in the config file
proxy_url="127.0.0.1"
proxy_port="8142"
privoxy_logfile="/usr/local/var/log/privoxy/logfile"
local_path=$( cd $( dirname "${BASH_SOURCE[0]}" ) && pwd )
privoxy_configfile="${local_path}/privoxy_configfile"

# Configs
echo ""
echo "========== Configs =========="
echo "proxy: ${proxy_url}:${proxy_port}"
echo "logfile: ${privoxy_logfile}"
echo "privoxy_configfile: ${privoxy_configfile}"
if [[ -n "${privoxy_debug_mode}" ]]; then
    echo "privoxy_debug_mode: ${privoxy_debug_mode}"
fi
echo "============================="
echo ""

if [[ "${privoxy_debug_mode}" = true ]]; then
    set -x
fi

# Ugly workaroud
# curl -O https://raw.githubusercontent.com/mackoj/privoxy-bitrise/master/privoxy_configfile

if [[ "${privoxy_debug_mode}" = true ]]; then
    ls
    networksetup -listallnetworkservices
fi

# install privoxy
brew install privoxy

# configure privoxy
ln -sfv /usr/local/opt/privoxy/*.plist ~/Library/LaunchAgents
privoxy_bin=$(/usr/libexec/PlistBuddy -c "Print:ProgramArguments:0" ~/Library/LaunchAgents/homebrew.mxcl.privoxy.plist)

# setup the proxy on OSX
sudo networksetup -setwebproxy "Ethernet" ${proxy_url} ${proxy_port}
eval "${privoxy_bin} ${privoxy_configfile}"

export http_proxy=http://${proxy_url}:${proxy_port}/

#verifing if privoxy is working properly
if [[ "${privoxy_debug_mode}" = true ]]; then
    ps aux | grep privoxy | grep -v grep
fi
privoxy_state=1
is_privoxy_working=$(ps aux | grep privoxy | grep -v grep | wc -l | awk '{print $1}')
if [[ ${is_privoxy_working} > 0 ]]; then
    privoxy_state=0
fi


if [[ "${privoxy_debug_mode}" = true ]]; then
    set +x
fi

# output all the logs
export PRIVOXY_LOG=${privoxy_logfile}
echo ""
echo "========== Outputs =========="
echo "PRIVOXY_LOG: ${PRIVOXY_LOG}"
echo "============================="
echo ""

exit ${privoxy_state}

From @viktorbenei on Thu Apr 21 2016 06:13:15 GMT-0400 (EDT)

Looks good, just a couple of notes:

  • If you want to reference a file which is inside the Step’s repository, you can do that in the Step the way @godrei mentioned (THIS_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" - $THIS_SCRIPT_DIR will be set to the absolute dir path of the script you call it from).
  • you should probably use set -e in your Bash script - by default Bash does not exit if there’s an error, if you add a set -e line to your script it’ll make Bash to exit right away when an error happens. You can turn it off with set +e and you can turn it on & off as many times as you want to. E.g. in your current script if brew install privoxy fails it’ll be just logged and ignored, Bash will continue with the rest of the script.
  • Dependencies: you should declare your brew dependencies (privoxy) in the step.yml, so that you don’t have to install it in your script, the CLI will check & install it if required
  • a simple export will not export the environment variable for other steps, those envs will only be available in the Step you call export. To export an env var so that other steps can access it you have to use envman (https://github.com/bitrise-io/envman)

From @mackoj on Thu Apr 21 2016 07:41:19 GMT-0400 (EDT)

Thanks for the advice @viktorbenei

From @mackoj on Thu Apr 21 2016 11:44:56 GMT-0400 (EDT)

Most of it is ready, I have to do more testing and improve documentation. I hope to share it to steplib soon.
I will be glad if some of you can beta test it and tell me what you think about it.

Cheers,

From @viktorbenei on Thu Apr 21 2016 12:19:15 GMT-0400 (EDT)

Awesome! We’ll try to make some time for it, hopefully next week - we’re a bit busy preparing for Xamarin Evolve :wink: