HTML Codes - Table of ascii characters and symbols. Analyst Scrapbook: A Bro script to extract artifacts from HTTP. The past few days I've been revisiting Bro (it has been awhile) for doing analysis and specific tasks when analyzing traffic dumps. Specifically of interest was carving out artifacts of interest (i.e., executables). Built into the base install of Bro is the "protocols/http/file-extract.bro" Bro script that allows you to redefine the "extract_file_types" variable to pull out files from HTTP sessions that match a specific MIME type. However, I wanted a more flexible Bro script to also extract out files that match magic bytes or are to a URL with a specific file extension - as well as having whitelisting functionality so that Windows Update, etc. are not constantly being stored to disk.
I finally have something that I'm fairly happy with and wanted to share with other budding Bro users. Side note: for me Bro has been best run against pcap files versus carving directly off the wire. Base/protocols/conn/main.bro — Bro 2.1 documentation. Blog: Filtering Logs with Bro. One of the best new features of Bro 2.0 is the logging framework. It gives you structured logs which are easily parsed for simplified log analysis. It also provides a nice abstraction between writing something to a log and handling that data before it is written to disk. I'll provide a very brief overview of the logging framework and then go into some filters that I've been helping people with lately. The logging framework in Bro 2.0 is based around sets of key-value pairs.
This alone was a huge step for Bro and helps bring it into the modern day since Bro logs now conceptually map neatly into all table and document store databases. To take it further, we wanted to separate the actions of sending data off to be logged and handling how that data is written to a data store (e.g. text files on disk). When data for a log is ready to be written out, log records are written to "Logging Streams" which can then be filtered, modified, and redirected with "Logging Filters". Ok, I lied. Exercise: Understanding and Examining Bro Logs. During the course of its normal operation, Bro produces a large volume of log files.
This series of exercises examines the Bro log output format, and highlights a few extremely useful utilities that can be used to extract data from and/or process this information. Bro summarizes each TCP and UDP connection as a single line in the conn.log. Because these connection summaries are quite detailed, you can extract plenty useful statistics from it. For the following two parts, use the log files generated from the trace 2009-M57-day11-18.trace.gz via bro -r 2009-M57-day11-18.trace.pcap.
Exercise List the connections by in increasing order of duration, i.e., the longest connections at the end. Solution awk 'NR > 4' < conn.log | sort -t$'\t' -k 9 -n The duration field records the number of seconds per connection. Find all connections that are last longer than one minute. awk 'NR > 4 && $9 > 60' conn.log Find all IP addresses of web servers that send more than more than 1 KB back to a client. Note. SampleCaptures. Sample Captures So you're at home tonight, having just installed Wireshark.
You want to take the program for a test drive. But your home LAN doesn't have any interesting or exotic packets on it? Here's some goodies to try. Please note that if for some reason your version of Wireshark doesn't have zlib support, you'll have to gunzip any file with a .gz extension. If you don't see what you want here, that doesn't mean you're out of luck; look at some of the other sources listed below, such as How to add a new Capture File If you want to include a new example capture file, you should attach it to this page (click 'attachments' in header above). Please don't just attach your capture file to the page without putting an attachment link in the page, in the format attachment:filename.ext; if you don't put an attachment link in the page, it's not obvious that the capture file is available. Other Sources of Capture Files General / Unsorted dns.cap (libpcap) Various DNS lookups. Capture network packets in Java with Jpcap.
If you want to capture network packets in your Java program, you might want to consider using Jpcap. Read this Java tip to learn how Jpcap works. If you want to capture network packets in your Java program, you'll need a little help because no parts of the core Java API give access to low-level network data. However, Jpcap is a Java API that provides you with this access on Windows or UNIX systems.
Jpcap isn't a pure Java solution; it depends on the use of native libraries. How Jpcap works Jpcap uses an event model to allow you to process packets. Public class JpcapTip implements JpcapHandler { public void handlePacket(Packet packet){ System.out.println(packet); } } In order to capture packets, you need to tell Jpcap which network device you want to listen with. String[] devices = Jpcap.getDeviceList(); Once you have a list of device names, you must choose one for listening: String deviceName = devices[0]; Jpcap jpcap = Jpcap.openDevice(deviceName, 1028, false, 10000);
Source Checkout - sjpcap - Simple, native java pcap file parsing library. Hypertext Transfer Protocol -- HTTP/1.0. HTTP Working Group T. Berners-Lee, MIT/LCS INTERNET-DRAFT R. Fielding, UC Irvine <draft-ietf-http-v10-spec-05.html> H. Frystyk, MIT/LCS Expires August 19, 1996 February 19, 1996 Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups.
Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. To learn the current status of any Internet-Draft, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). Distribution of this document is unlimited. Abstract Table of Contents 1. 1. 1.1 Purpose 1.2 Terminology connection message request entity. Fast, easy, realtime metrics using Redis bitmaps | At Spool, we calculate our key metrics in real time. Traditionally, metrics are performed by a batch job (running hourly, daily, etc.). Redis backed bitmaps allow us to perform such calculations in realtime and are extremely space efficient. In a simulation of 128 million users, a typical metric such as “daily unique users” takes less than 50 ms on a MacBook Pro and only takes 16 MB of memory.
Spool doesn’t have 128 million users yet but it’s nice to know our approach will scale. We thought we’d share how we do it, in case other startups find our approach useful. Crash Course on Bitmap and Redis Bitmaps Bitmap (aka Bitset) A Bitmap or bitset is an array of zeros and ones. Population Count The population count of a Bitmap is the number of bits set to 1. Bitmaps in Redis Redis allows binary keys and binary values. A simple example: Daily Active Users To count unique users that logged in today, we set up a bitmap where each user is identified by an offset value.
Optimizations Sample Code. What Every Developer Should Know About URLs. I have recently written about the value of fundamentals in software development. I am still firmly of the opinion that you need to have your fundamentals down solid, if you want to be a decent developer. However, several people made a valid point in response to that post, in that it is often difficult to know what the fundamentals actually are (be they macro or micro level).
So, I thought it would be a good idea to do an ongoing series of posts on some of the things that I consider to be fundamental – this post is the first instalment. Being a developer this day and age, it would be almost impossible for you to avoid doing some kind of web-related work at some point in your career. That means you will inevitably have to deal with URLs at one time or another. We all know what URLs are about, but there is a difference between knowing URLs like a user and knowing them like a developer should know them. The Structure Of A URL This is easy, starts with HTTP and ends with .com right :)? Related. Untitled. Basic Bro Language References. Finding simple examples of Bro language features is somewhat difficult: the scripts that come packaged with Bro are written by experts in the language and are quite idiomatic.
Here are some of the basic Bro language references I've found so far. In upcoming blog posts, I'll show some "Baby Bro" that is even more basic than these examples. From Ryesecurity by Scott Runnels(@srunnels):Solving Network Forensic Challenges with Bro, Part 1Solving Network Forensic Challenges with Bro, Part 2Solving Network Forensic Challenges with Bro, Part 3Logging YouTube Titles with Bro Justin Azoff's (@JustinAzoff) Bro Presentation on Github The Official Bro Workshop 2011 Pages The Bro Language Cheat Sheet.