greptrick.com

Incoherent ramblings of an InfoSec professional

PFSense, Netflow and ELK w/geoip

Several months ago I started working with the ELK stack (elasticsearch, logstash, kibana) for use with bluecoat proxy logs. I have been running pfsense at home for quite sometime and decided it would be nice to get some data pulled out of it, why not with netflow. As with everything else there are pieces of stuff all over the interwebs, but nothing that pulled it all together for me to use.

Here is a simple breakdown of the steps.

Configure pfsense to pass flow data
install java
install elasticsearch – research optimizations for single node.
install kibana 4
install logstash
install nginx
configure logstash
configure kibana – secure
configure nginx
configure elasticsearch – secure
configure mappings for flow
build dashboards in kibana

Here is the base setup.

Debian 8.1 64bit running on ESXi
– 2 vCPUs
– 8GB Ram
– 60G Storage

A. Setup PFSense to collect and pass flow data

Install softflowd package that is available for pfsense.

Screen-Shot-2015-07-13-at-3.55.11-PM.png

Services -> softflowd select “Interface, Host “ip of ELK box”, Port “9995” (will be configured later in logstash config)

Screen Shot 2015-07-13 at 9.44.07 PM

B. Install and configure ELK – a good chunk with modifications was taken from this DigitalOcean article – https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04

1. Install Java (latest version)

Add repo list and update –

 sudo add-apt-repository -y ppa:webupd8team/java && apt-get update
 

Install java8

 sudo apt-get -y install oracle-java8-installer
 

2. Install Elasticsearch NOTE:I just used the latest (1.6.0) as of the time of writing this.

Download GPG key for apt.

    wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
 

Add to repo list and update.

    echo 'deb http://packages.elasticsearch.org/elasticsearch/1.6/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.list
 

Install

    sudo apt-get -y install elasticsearch=1.6.0
 

Edit /etc/elasticsearch/elasticsearch.yml and set network.host: localhost

3. Install Kibana 4 NOTE:Latest as of this writing 4.1.1

  wget https://download.elasticsearch.org/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
  tar -zxvf kibana-4.1.1-linux-x64.tar.gz

Move the code over to someplace that seems normal

sudo mkdir -p /opt/kibana && sudo cp -R ~/kibana-4*/* /opt/kibana/

Go head and install this git code that will setup kibana as a service.

cd /etc/init.d && sudo wget https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/bce61d85643c2dcdfbc2728c55a41dab444dca20/kibana4
sudo chmod +x /etc/init.d/kibana4

At this point edit the kibana config in /opt/kibana/config/kibana.yml and set host: “localhost”

4. Install Logstash NOTE:Latest as of this writing 1.5.2

Add repo and update.

  echo 'deb http://packages.elasticsearch.org/logstash/1.5/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list && sudo apt-get update

Install Logstash

  sudo apt-get install logstash

5. Install nginx

We will use nginx as a reverse proxy to get to the ELK instance. Kibana and Elasticsearch do not offer much in the way of security so we have to lock them to only being accessible from localhost.

apache2-utils only necessary if htpasswd is desired to help secure the kibana instance.

  apt-get install nginx apache2-utils 

6. Configure nginx reverse proxy

Replace /etc/nginx/sites-available/default with the following.

server {
    listen 80;

    server_name <servername>;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

7. Configure Logstash to accept traffic and create geoid data.

First get the geoip data, this is updated monthly so a cron is probably a good idea. This will create the lookup database for logstash.

cd /etc/logstash && sudo curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz" && sudo gunzip GeoLiteCity.dat.gz

Create /etc/logstash/conf.d/logstash-netflow.conf place the following in the file.

input {
   udp {
     port => 9995
     codec => netflow
     type => netflow
   }
}
filter {
  geoip {
    source => "[netflow][ipv4_src_addr]"
    target => "src_geoip"
    database => "/etc/logstash/GeoLiteCity.dat"
  }
  geoip {
    source => "[netflow][ipv4_dst_addr]"
    target => "dst_geoip"
    database => "/etc/logstash/GeoLiteCity.dat"
  }
}
output {
  stdout { codec => rubydebug }
  if ( [host] =~ "10\.100\.1\.1" ) {
    elasticsearch {
      index => "logstash_netflow-%{+YYYY.MM.dd}"
      host => "localhost"
    }
  } else {
    elasticsearch {
      index => "logstash-%{+YYYY.MM.dd}"
      host => "localhost"
    }
  }
}

Parts of this are for future growth, where [host] =~ .

8.

Start everything up and set to start on boot

    sudo systemctl enable elasticsearch.service
    sudo systemctl enable logstash.service
    sudo systemctl enable kibana4.service
    sudo systemctl enable nginx.service
    sudo systemctl start elasticsearch.service
    sudo systemctl start logstash.service
    sudo systemctl start kibana.service
    sudo systemctl start nginx.service
 

Everything now should be working, check your logs /var/log/logstash/logstash.log and /var/log/elasticsearch/elasticsearch.log – once you get data pushed into logstash ES should create an index in /var/lib/elasticsearch/elasticsearch/nodes/0/indicies

Now we just need to connect to Kibana and configure some stuffs

9. Create geo_point mappings

Once we are sure that data is flowing we have to create static mappings for the geo-location data that is created by logstash. ES will do dynamic mappings but Kibana won’t see them correctly to create a map of the data. Below worked on my setup, but it took me a while to get it right.

This will apply the “logstash_netflow_template” to any index named “logstash_netflow*” our index is date based (logstash_netflow-YYYY.MM.DD) so this will cover every index created going forward.

root@Logstash:~# curl -XPUT http://localhost:9200/_template/logstash_netflow_template -d '
{
  "template" : "logstash_netflow*",
  "mappings" : {
    "netflow" : {
      "properties" : {
        "dst_geoip" : {
          "properties" : {
            "location" : {"type":"geo_point"}
          }
        },
       "src_geoip" : {
         "properties" : {
           "location" : {"type":"geo_point"}
         }
       }
     }
   }
 }
}'

10. That should be it for configs, the rest can be done inside Kibana.

Go to kibana http:// once ELK loads, go to Settings -> Indices

Create New Index Pattern, select “Use event times to create index names” set the Index name or pattern to [logstash_netflow-]YYYY.MM.DD – You should get pattern match if you index is created properly, select Time-field name of “@timestamp” then just hit “Create”

Screen-Shot-2015-07-13-at-9.14.54-PM.png

I will save the rest of the Kibana setup for another post or for the reader to struggle through, here are some shots of my dashboard as it sits using the data collected above.

Screen Shot 2015-07-13 at 9.25.43 PM

Screen Shot 2015-07-13 at 9.25.59 PM

Screen_Shot_2015-07-13_at_9_26_18_PM

Screen_Shot_2015-07-13_at_9_26_47_PM

There are a lot of performance tuning for ELK that I will talk about in another post. This will work for a small installation. This is running at my home with average usage of about 1000 events/min. In a larger environment you will likely need to separate your collectors (logstash) from your elasticsearch instances, and tune your ES. Rule of thumb, ES_HEAP_SIZE = half of your system ram but never more than 32g. If you need more than that spin up a second instance of ES.

Eastern Iowa InfoSec Meetup

About a month ago Ryan Stillians came to me with an idea to do a local meet up for information security professionals in the area. Since I had sorta had this same idea but never got it off the ground we decided to work together to get something off the ground. This is designed to be a monthly meet up group with some additional things if we have a topic that we really want to dig into. Most meetings will take place between Iowa City and Cedar Rapids and hopefully back and forth. The desire is to create a community in the area where info sec professionals can get to know others. Hopefully this will keep the good momentum we saw from the first meeting.

First meeting was great, second meeting is tonight (12/9) where we are planning some member introductions.

Check out our meet up site http://www.meetup.com/Hawkeye-InfoSec/

or via twitter @HawkeyeInfoSec

 

Java – Can’t Uninstall? Whitelist it?

As with most sizable organizations it is near impossible to uninstall or completely disable Java which sent us on a hunt for a feasible way to contain Java based attacks. What we came up with was restricting it to run only in trusted zones. This worked for APPLET tags when encountered in IE. 

What this does is block any applet from running if it is not part of a trusted internet zone. First thing is to identify all the internal trusted zones and add them. Next allow the user to trust their own zones. Most of the time it seemed they knew when there was an applet they wanted to run. To enable this there is a simple registry change value 1C00 in 

 HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Zones\3 to a value of 0
*Note: (original 10000). 

This will prohibit Java from running in the “Internet Zone.” Now for internal sites you can just whitelist them as a “Trusted Zone” for java to run properly. Of course this can be done via GPO for all internal sites and if there are some identified external sites that java is required. 

For some fun stats, an enterprise of 15,000 endpoints went from ~1.5 take aways per day down to about 1-2 per month due to Java a java drive by style attacks.

For most organizations updating java is a Herculean effort. So the whitelisting method from within Windows can be a viable alternative that can be quite effective. In recent months Oracle has released Java 1.7U40 which includes whitelisting. This is nice because it will work for browsers other than IE but odds are if you can get to 7U40 you have a good handle on patching anyway so this is less of an issue (except for 0-day). Java calls their implementation “Deployment Rule Sets” (DRS). 

A DRS is just a XML configuration file listing the location or hash of a jar and the action to take. You can bypass some of the pop-ups (some pop-ups can’t be disabled such as JAR unsigned), you can flat out block the jar or run the default actions as you would if DRS wasn’t defined. The XML file is parse sequentially so place your allowed jars at the top of the file and place a catch all block rule at the bottom. 

Deployment of the rule set is as simple as packing in a signed (from a trusted 3rd party) jar file named DeploymentRuleSet.jar and deploying to the endpoints to be controlled. 

Java based whitelisting is a very powerful feature but it is limited. With MSFT based whitelisting users can individually whitelist java for their own uses, but it is limited only to IE. Java based on the other hand does stop end users from whitelisting however it is a larger effort to whitelist and you either have to manage many lists for individual or groups of users or you have to whitelist sites for everyone, package and re-push.

Also, deploy EMET

 

Sources:

Oracle docs on the setup.

http://docs.oracle.com/javase/7/docs/technotes/guides/jweb/deployment_rules.html
https://blogs.oracle.com/java-platform-group/entry/introducing_deployment_rule_sets

Push TrustedSites via GPO

http://www.makeuseof.com/tag/configure-trusted-sites-internet-explorer-group-policy/

Java Whitelisting from MSDN

http://blogs.msdn.com/b/ieinternals/archive/2011/05/15/controlling-java-in-internet-explorer.aspx

Book Review: The Practice of Network Security Monitoring

NewImage

 

Earlier this year Richard Bejtlich (@taosecurity) was on PDC http://pauldotcom.com/wiki/index.php/Episode327 Since that time his book as been released. The Practice of Network Security Management from NoStarch (http://nostarch.com/nsm). Richard is an individual who knows what he is talking about when it comes to NSM, having run a CIRT for the Air Force and GE and now resides as the CSO for Mandiant. Richard has also authored The Tao of Network Security Monitoring and Extrusion Detection. 

This an excellent resource for beginners to NSM or people starting to do NSM in their environment and it is not intended for experienced NSM practitioners. It begins with a very good overview of what NSM is and what constitutes NSM and the sort of data that needs to be collected. Gives some good examples of how to place sensors and caveats found in large organizations with NAT and the like. Then to tie the types of devices into a tool, Richard introduces a very good set of Open Source tools like, Bro (bro.org), Snort, Squil, Snorby, ELSA etc. that are part of the Security Onion distribution created by Doug Burks (@dougburks).

The first section of the book is dedicated to in the installation of Security Onion and configuration of the tools that go along with it so that the reader could spin up their own instance and learn along with the book. If you are in anyway experienced with Linux installations you can skip most of this section, it was written so folks without much exposure can install SO. Once that is taken care of he builds on the tools contained within SO. When he can he shows multiple ways and tools to achieve the same thing, for example using tshark and the GUI version of Wireshark to analyze packets. He gives real world examples of data that tripped things like Snort to help understand the workings of the tools. 

After the basics and tools are introduced it dives into using NSM to monitor a network. Areas like phases of an investigation and what data might help identify success vs attempt in an incident. He discusses incident classification which is one place I think CIRTs miss. The best part in my opinion of the whole book is in chapters 10 and 11. Once everything is introduced and a good understanding of the tools are in place Richard walks through 2 sample breech investigations and takes a step by step approach on the analysis done, data and tools used to identify the extent of a breach. 

This is a very well written technical book. This is a very easy and quick read for its size, provides many examples and screen shots to give readers a good understanding of how to use the tools in the book. I would recommend this for anyone getting into the field of incident response who doesn’t have a great understanding of NSM, newcomers or perhaps less technical Managers.

-Greg Hetrick @gchetrick

Going Live on pauldotcom soon to talk SR

Going Live on pauldotcom soon to talk SRP. Grab a beer and join in.

SSH over Stunnel for IDS evasion

A few weeks ago (Episode 329 http://pauldotcom.com/wiki/index.php/Episode329#Tech_Segment:_Free_Amazon_Socks_Proxy_by_Allison) Allison gave a great segment on avoiding firewalls using port forwarding and SOCKS proxy via ssh with a server on port 443 using free Amazon AWS instance. Something struck me:

1) you could have a proxy block SSH traffic going over 443.

2) you could haven IDS detect and, if inline, block since any IDS will detect SSH over non-standard ports. 

So how do we fix this problem? Encapsulate SSH traffic inside SSL of course. In comes stunnel! Stunnel creates a SSL tunnel to pass almost any traffic through it. All you need is the same AWS instance that Allison talked about with stunnel installed and a client on the other end also with stunnel.

The setup:

yum or apt-get install stunnel – Both server and client (there is a macport of stunnel as well as Android and Windows installers at stunnel.org)

Server side configuration, create file stunnel.config with the contents below:

cert=/path/to/stunnel.pem
pid=/tmp/stunnel.pid
[ssh]
accept = <serverip>:443
connect = 127.0.0.1:22

Create Self Signed certificate.

Create a key

 $ openssl genrsa 1024 > stunnel.key

Generating RSA private key, 1024 bit long modulus
………………………++++++
……..++++++
e is 65537 (0x10001)

Generate the self signed certificate.

$ openssl req -new -key stunnel.key -x509 -days 1000 -out stunnel.crt
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
—–
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

Next up create the PEM file which just contains the key and the crt contents

$ cat stunne.crt stunnel.key > stunnel.pem

Now you can start the tunnel — in Debian there is a perl wrapper for stunnel4 that is /usr/local/bin/stunnel — when I envoked stunnel this way the tunnel would not start. When I bypassed the wrapper and called stunnel4 directly it worked fine.

$ stunnel4 stunnel.config

verify with netstat that the port is running (netstat -tanp) or you can test it with an openssl command: $ openssl s_client -connect <ip>:443

Now over on the client side copy over the .pem certificate from the server and place it somewhere. Then create your client configuration file stunnelclient.config

cert=/home/ghetrick/stunnel/stunnel.pem
pid=/tmp/stunnel.pid
client=yes
[ssh]
accept=2200
;protocol=connect
;protocolHost=<ip:port>
;protocolUsername=<username>
;protocolPassword=<pass>
connect=<ip: port>

*If you need to go through a proxy you can uncomment the “protocol” lines and fill out the information accordingly. Where protocolHost is the server you want to connect to, and connect becomes the ip and port of the proxy device. If there is no proxy present the connect line is the ip and port of the remote host to connect to.

Now fire up stunnel on the client machine

$ stunnel4 stunnelclient.config

netstat -tapn to verify it is running on the assigned local port, in this example it is 2200

Now you can simply use ssh to the localhost

$ ssh -p 2200 localhost

and you are on the remote server, all hidden in nice SSL packets. I was able to run through a web proxy on port 443 with this config and did not trip signatures on the IDS that was otherwise tripping without stunnel in the middle.

You should be able to follow Allison’s documentation for getting port forwarding working properly for full enjoyment.

Go forth and evade.

-@gchetrick

Thwarting Client Side Attacks with Software Restriction Policy

A few weeks ago I started looking at Windows Software Restriction Policy (SRP) and using it to stop client side attacks. This is going to go over some of the options, setup and the results once enabled.

SRP is easy to setup via Group Policy Object (GPO). Inside GPO editor create New Software Restriction Policy. Once create the default will be setup. You can look around to see basic options. Here is my tested setup.

Enforcement: Select “All Software files” and “All users except local administrators”

Enforcement Properties

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Under Designated File types: Remove type LNK – this will make sure that shortcuts placed outside of the designated execution directories will run. When I initially tested what I thought would work none of the shortcuts on the toolbar or desktop would launch an application and I found this to be the issue.

FileTypes

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Ignore trusted publishers, this is used if we are limiting applications based on the certificate authority.

Select “Additional Rules”

The default execution directories will be selected.

%HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SystemRoot%
%HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\ProgramFilesDir%

Since mine is 64bit Windows I added

%HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\ProgramFilesDir (x86)%

Security level for these are all going to be “Unrestricted” I want them to be able to execute as normal.

Now back under “Security Levels” the default setting is Unrestricted, since we are changing users over to defined execution directories I want to set anything not specifically allowed in the Additional Rules section to “Disallowed.” So we change the default to Disallowed.

Save this and run gpupdate /force on the target machine.

Now to test a client side attack using SET. I am going to use the java attack method. 1 -> Social-Engineering Attacks, 2 -> Website Attack Vectors, 1 -> Java Applet Attack Method, 1 -> Web Templates, 1 -> Java Required, 2 -> Windows Reverse_TCP Meterpreter, 16 -> Backdoored Executable – Enter port of listener (default 443)

Fire it up and wait till it starts the payload handler.

SET Launch

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Once the handler is started you are ready to test the attack. Go ahead and run the unsafe java applet.

Java Applet

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

You will notice that the the site is responding but the java applet is unable to execute the payload.

SET Failure

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

After attempting this and being successful, I tried running SET with PowerShell Injection and to my surprise the attack succeeded. I realized with PowerShell the payload was running from the C:\Windows\sysWOW64\WindowsPowerShell directory which by default is explicitly allowed. To defeat this attack I added the path to the list of Additional Rules and set it to “Basic User”, retested the attack with PS Injection and the attack failed as expected. I tested this with multiple payloads and encoding methods and everyone of them did not result in a successful attack. On 32bit systems PowerShell is located in C:\Windows\System32\WindowsPowerShell\v1.0\ so that directory will need to be accounted for as well.

I ran two other tests, the first was using EXE embedded PDF and an older version of Adobe Reader (9.3). SRP was able to successfully stop this attack.

Finally I tested a physical attack using a USB Rubber Ducky Human Interface Device (HID) from the folks over at hak5 (www.hak5.com). I used a great little payload generator found over on google code (https://code.google.com/p/simple-ducky-payload-generator/ ) It is pretty slick and simple, I used a meterpreter powershell injection payload that didn’t attempt to elevate privileges. SRP was able to successfully stop this attack. If the user had admin privileges and entered in creds in the UAC window it would have worked since I allow Local Admins unrestricted access.

In Production the are likely other directories where code needs to execute, those will need to be added to the allow list. As the config is done, administrators will be able to bypass these rules for installation of software etc. Administrators will also need to ensure that ACLs are properly set since a curious user could move executables into the approved directories and run them. While this is like a bit tough to implement in a very large organization this is a very effective method for stopping client side attacks.

To find other executable directories in use in your environment enable SRP with defaults (fully unrestricted) and set the following registry key:

“HKLM\SOFTWARE\Policies\Microsoft\Windows\Safer\CodeIdentifiers”
String Value: LogFileName, <path to log file>

This will log the executable and the directory it was run from a little data mining can determine were applications need to execute from. Also Inventory Collector from Application Compatibility toolkit can assist in this task.

Google Drive Backup Script

A few weeks back I was looking for a solution to push backup files off a linux box and push them to google drive. All I could seem to come across was apps to sync with google drive on linux. While this would have worked, I didn’t really need nor want any of the files in my Google Drive on a linux server. So I decided to write my own. This is my first foray into a python application of sorts, I have been doing development in PERL for most of my career. This is also my first attempt at interfacing with an API, and I couldn’t have picked a better project since the Google API docs are excellent!

The code is published on my github account. You will need your own developer API key for it to work since sharing publicly isn’t an option and I haven’t come up with a way to securely pull the API key.

https://github.com/gchetrick/gDriveBackup

Currently you need to create a file with the API information to connect to google drive. Once you have that, initialize with google to proved authorization to your google drive and it will then allow you to specify a file or directory to push up.

Keep in mind I am a admin not a developer so the code will be written as such. I am working on going back to clean it up.

Features in the works:
Define and create a directory in Google Drive to upload to
Add ability to encrypt perhaps using TrueCrypt or gpg (of course this can be done prior to upload too)
Resume uploads
Do proper verification that upload completed successfully.

Mujjo – Premium MacBook Air Sleeve

I recently ordered a Mujjo sleeve for my MacBook Air. I must say this has to be once of the nicest sleeves of any type I have ever seen. It is a bit expensive weighing in at ~$92 shipped but it is well worth it. 

They make them for many different items, I am strongly thinking about one for my iPad. Only reason I haven’t pulled the trigger is that I am thinking of getting an iPad Mini, but I am holding out for the Mini Retina.

Check it out here!

http://www.mujjo.com/the-sleeves-collection/macbook-sleeves/macbook-air-13-sleeve-brown-wool-felt

Crowd Strike? Seriously?

This is a bit of a quick rant. I heard about this last week but didn’t get around to commenting. Apparently last week at RSAC CrowdStrike decided to throw ethics aside for showmanship by bringing down the Kelihos botnet on stage in front of a crowd.

Why? How long were they siting on this just so they could do in front of a bunch of people and press?

While I applaud the work they did do, I wonder about the people who got their creds stolen and bank accounts raided between when they knew how to take it down and when they actually did on stage. This seems completely irresponsible and shaming to see what some folks in the InfoSec industry are headed to.

Read more about the take down at Threatpost.

http://threatpost.com/en_us/blogs/latest-kelihos-botnet-shut-down-live-rsa-conference-2013-022613