background preloader

AWS-Unix-Shell-Script Tricks

Facebook Twitter

ShellCommandActivity - AWS Data Pipeline. Runs a command or script.

ShellCommandActivity - AWS Data Pipeline

You can use ShellCommandActivity to run time-series or cron-like scheduled tasks. When the stage field is set to true and used with an S3DataNode, ShellCommandActivity supports the concept of staging data, which means that you can move data from Amazon S3 to a stage location, such as Amazon EC2 or your local environment, perform work on the data using scripts and the ShellCommandActivity, and move it back to Amazon S3. In this case, when your shell command is connected to an input S3DataNode, your shell scripts to operate directly on the data using ${INPUT1_STAGING_DIR}, ${INPUT2_STAGING_DIR}, etc. referring to the ShellCommandActivity input fields. An A-Z Index of the Bash command line. Commands marked • are bash built-ins Many commands particularly the Core Utils are also available under alternate shells (C shell, Korn shell etc).

More bash commands: Linux Command Directory from O'Reilly, GNU CoreUtils.SS64 bash discussion forumLinks to other Sites, books etc. ShellCommandActivity - AWS Data Pipeline. Shell - Script to monitor folder for new files? Linux - How to continuosly monitor the directory using dnotify /inotify command. Uploading and processing data with inotify-tools - post in A Foo walks into a Bar... - blog by Pavel Shved - I'm not a muscleman.

Uploading and processing data with inotify-tools - post in A Foo walks into a Bar... - blog by Pavel Shved -

In fact, I'm kind of a wimp. But given all this, my weight is 15 kilograms greater than I should have with my height. Time to lose some. I've been logging my weight for quite a time, but I realized that logging alone doesn't help. I need other people to nudge me, so I would be more ashamed of my condition. And, of course, since I'm a geek, the process of displaying the graph should be made as automated as possible. Nearly every morning I weigh and log the result in the text file. What is inotify Inotify is an interface to Linux kernel; its purpose is to watch and report when certain filesystem events happen. Just like Berkeley sockets, inotify calls create file descriptors, which can then be watched by usual select() calls. A list of events that may be watched includes accessing file, modifying file, moving files and modifying metadata.

More info you can get in man 7 inotify and on Wikipedia. How to use inotify-tools to trigger scripts on filesystem events - In my last example of Bash if statements we created a backup script which would run “rsync” as per the time set in a cron.

How to use inotify-tools to trigger scripts on filesystem events -

But wouldn’t it be nice if we could run our “rsync” script only if there was a change to the files in our directory instead of running a cron every hour despite no change? With inotify-tools you can do just that. What is inotify-tools? Inotify tools are a set of command line programs based on inotify a Linux kernel (2.6.13 or later) feature which can be used to monitor filesystem events. Installing inotify-tools Software versions : inotify-tools.x86_64 3.14-1.el6 CentOS 6.5 Linux kernel 2.6.32-042stab085.20 #Add the epel repo [leo@linux-vps ~]$sudo rpm -Uvh #Install inotify-tools [leo_g@vps ~]$ sudo yum install inotify-tools Inotify-tools has two commands Inotifywait This command simply blocks for inotify events, making it appropriate for use in shell scripts.

Inotifywatch inotifywatch collects filesystem usage statistics and outputs counts of each inotify event. source. 30 Handy Bash Shell Aliases For Linux / Unix / Mac OS X. An alias is nothing but shortcut to commands.

30 Handy Bash Shell Aliases For Linux / Unix / Mac OS X

The alias command allows user to launch any command or group of commands (including options and filenames) by entering a single word. Use alias command to display list of all defined aliases. You can add user defined aliases to ~/.bashrc file. You can cut down typing time with these aliases, work smartly, and increase productivity at the command prompt. More about aliases The general syntax for the alias command for the bash shell is as follows.

Task: List aliases Type the following command: alias Sample outputs: alias .. By default alias command shows a list of aliases that are defined for the current user. Task: Define / create an alias (bash syntax) Append Current Date To Filename in Bash Shell. How do I append current date (mm_dd_yyyy format) to a filename (e.g., backup_mm_dd_yyyy.sql) under UNIX like operating systems?

Append Current Date To Filename in Bash Shell

To get the current date in mm_dd_yyyy format use the following date format syntax: date +"%m_%d_%Y" You can store this to a variable name: now=$(date +"%m_%d_%Y") now=`date +"%m_%d_%Y"` Finally, you can create a filename as follows: now=$(date +"%m_%d_%Y")echo "Filename : /nas/backup_$now.sql" Sample outputs: