background preloader

Aws

Facebook Twitter

Как обрабатывать терабайты данных в 1000 потоков на PHP — Hadoop/MapReduce / Блог компании 1С-Битрикс. Всем привет!

Как обрабатывать терабайты данных в 1000 потоков на PHP — Hadoop/MapReduce / Блог компании 1С-Битрикс

Уже слышали про Bigdata? Ну да, веб растет, данных становится больше и их нужно держать под контролем и периодически анализировать. Базы данных — лопаются под нагрузкой, реляционная теория не совсем справляется с задачей, нужно решение. Маркетинг активно давит сверху, а железо острыми углами — снизу и попахивает суицидом. В этом посте постараюсь дать конкретные работающие рецепты и куски кода с краткими теоретическими выводами, как же обрабатывать >=терабайты в >=1000 потоков на PHP. Однако, если вдруг стало подташнивать и закружилась голова, можно дальше не читать — а полюбоваться на прекрасных птичек и забыть о вышенаписанном. Как обычно делается Как обычно бывает в вебе. Securing your AWS Infrastructure with IAM.

When developing a new product and finding that product market fit every team needs to move fast.

Securing your AWS Infrastructure with IAM

Especially startups, as the whole future of the company is dependent on quickly finding the people who pay for your product. Amazon Web Services has been an incredible tool for startups and other teams to build their application and infrastructure quickly. These teams often have a stronger background in development than proper system operations. AWS provides great tools to handle keys, authentication and security, but many teams haven’t looked into them, as more pressing issues often come up. We want to introduce a couple of concepts and features that will help you make your infrastructure more secure, while taking only minimal effort.

Identity and Access Management (IAM), the heart of AWS security IAM, Identity and Access Management, is the core of the AWS security system. Make a New Year Resolution - AWS Security Blog. Deep Dive into Amazon ElastiCache. Connection Overheads Amazon ElastiCache node (MemCached engine) uses a connection buffer per TCP connection to read/write data out over the network.

Deep Dive into Amazon ElastiCache

This is not a problem when there are few concurrent connections to an Amazon ElastiCache Node, whereas when you get into hundreds of thousands of connections with hundreds of Amazon ElastiCache nodes, this adds up to connection memory overheads. Example: On each Amazon ElastiCache node, the memory made available for storing cache items is the total available memory on that cache node minus the memory used for connections and other overhead (like TCP connection buffer). This overhead value can be configured using the memcached_connections_overhead parameter in Amazon ElastiCache. For example, a cache node of type cache.m3.2xLarge has a max_cache_memory of 29600 MB. Benchmarking High Performance I/O with SSD for Cassandra on AWS. Benchmarking High Performance I/O with SSD for Cassandra on AWS.

Beanstalk

Installing Percona XtraDB Cluster on Ubuntu — Percona XtraDB Cluster Documentation. This tutorial will show how to install the Percona XtraDB Cluster on three Ubuntu 12.04.2 LTS servers, using the packages from Percona repositories.

Installing Percona XtraDB Cluster on Ubuntu — Percona XtraDB Cluster Documentation

Installation Percona repository should be set up as described in the Percona apt Repository guide. Following command will install Percona XtraDB Cluster packages: $ apt-get install percona-xtradb-cluster-server-5.5 percona-xtradb-cluster-client-5.5 percona-xtradb-cluster-galera-2.x When these two commands have been executed successfully on all three nodes Percona XtraDB Cluster is installed. Note Debian/Ubuntu installation prompts for root password, this was set to: Passw0rd. Configuring the nodes Individual nodes should be configured to be able to bootstrap the cluster.

Configuration file /etc/mysql/my.cnf for the first node should look like: For the first member of the cluster variable wsrep_cluster_address should contain empty when the cluster is being bootstrapped. After this, first node can be started with the following command: Part 4: AWS High Availability Patterns : Database Tier. High Availability @ Database Layer Data is the most valuable part of any application and designing High Availability for the Database Layer is the most critical priority in any Fault tolerant architecture.

Part 4: AWS High Availability Patterns : Database Tier

To avoid Single Point of Failure in the Database layer it is a common practice to launch the Multiple Database server's in Master-Slave replication or Cluster mode. Now let us see some of the common architecture practices and considerations for the same in AWS: Master Slave Replication: We can configure 1 MySQL EC2 as master and 1 or more MySQL EC2 as Slaves in this architecture.

Scaling Symfony 2 with AWS OpsWorks - Alexfu. During the last complete rebuild of Goalshouter we needed to redesign our infrastructure to handle the big traffic we were expecting.

Scaling Symfony 2 with AWS OpsWorks - Alexfu

Our biggest difference between an everyday site is that 80% of the traffic is concentrated in some hours of sunday because most of the soccer matches are at that time. We needed a solution that was being able to scale fast to a large number of servers. According to New Relic on sunday afternoon the requests per second increases by 20x in a couple of minutes. Our match processing tool is a bit resource intensive and we need enough power to let teams track their matches and people follow them.

Technical Blog for Software Enthusiasts. Finally we have reached the end of this saga of customizing AWS Elastic Beanstalk using suite of Linux Bash scripts.

Technical Blog for Software Enthusiasts

This post contains a link to a downloadable zip file which contains all the scripts and here are the step by step instructions on using it. 1. Download the zip file and unzip its contents […] Let us turn our attention to the development environment which is what we extensively use on a daily basis at hudku.com. I strongly recommend one and all to make use of the Amazon’s AWS Free Usage Tier, fire up EC2 micro instance and give all the AWS services a spin.

You could be using the […] In this post we are going to deploy a new version of the application onto an existing AWS Elastic Beanstalk environment and it could be development, staging or the production environment. Configuration files and scripts for the blog post: Faster Rails 3 deployments to AWS Elastic Beanstalk. Faster Rails 3 deployments to AWS Elastic Beanstalk ← Vinicius Horewicz. Web applications deployments cannot be slow.

Faster Rails 3 deployments to AWS Elastic Beanstalk ← Vinicius Horewicz

The definition of slow, of course, is relative. I like to think slow is anything that breaks the flow, that takes long enough to make me lose focus of what I am doing. Recently I moved a Rails 3 app from Heroku to Amazon Elastic Beanstalk (EB). Deploying to Heroku was not showing the best performance results, but to EB was taking enough time to make me lose my patience. After understanding how EB works, identifying bottlenecks, and caching things; I was able to reduce the time to deploy to near one minute. EB: Behind the scenes EB allows deployments in a Heroku-esque way simply executing git aws.push. When EB receives the latest version of your app, it packs the source files in a zip archive, uploads to a S3 bucket under your account, and deploys to the EB environment. Inspecting the EC2 plumbing Analyzing the file /var/log/eb-tools.log, we can see the deployment has hooks and scripts associated with it.