Попался хороший коммент про pros and cons Редшифт, староват, но уж очень в точку мущщина написал - не прибавить, не убавить.
Комент к статье
https://www.quora.com/What-are-the-pros ... n-Redshift" onclick="window.open(this.href);return false;
Вот этот
Pros:
Scarily fast. We run a 6 node SSD cluster (6 of the small nodes), and we can run aggregations on hundreds of millions of rows in a few seconds. The most crazily complex queries we've written return in less than a minute usually. Our median response time is 2 secs. Tip: run SSD clusters if you need the disk IO but don't need the storage space. It's worth it.
Cheap - it's around a 10th of the cost of competitors.
It's AWS - which works great if you're already on it, like we are at Hailo.
It's SQL - so you can pretty much cut and carve your data as you like. The fork of postgres only takes a few commands out. We've always been able to do what we want. We can go from thinking of an idea, to writing a query, to answering questions in a couple of minutes. I doubt we'd be able to do this with Hadoop/Hive.
Scalability - it's horizontally scalable. Need it to go faster? Just add more nodes.
Cons:
Doesn't enforce uniqueness. This means you'll have to be very diligent about data hygiene. If you're running distributed systems that write to Redshift, you'll probably have to use some caching system like Redis to check if you've written the data to the database already.
Very fast, but not fast enough for most web apps. This means you'll have to pull data into a caching layer, or a vanilla postgres instance if you want to serve web apps with data from Redshift.
Loading data - the fastest way to do this to take advantage of Massively Parallel Processing (what a great buzzword) is to load in from S3. Alternatively, you can do JDBC inserts, but these can't be parallelised. Ties you into the AWS stack more.
Difference from vanilla SQL - I don't necessarily view this as a bad thing, but you'll have to get your head around how to optimise the distribution of your data around the nodes using DISTKEYS, and check that you have appropriate compression on your columns. You also can't alter the datatypes of a column in a table after they've been set, which can be annoying.
Summary:
It's awesome. We've been nothing short of ecstatic. Ping me if you need any pointers.
PS. Выделенное жирным было для меня полным шоком. Нашли недавно немало неожиданных дупов, при том что в нашем ETL процессе присуствует dedup логика по unique key с md5 cache
PSS. Интересная задача - джойнить timeseries like data in warehouse (Redshift) with stream data для принятия real time decisions. Примеров где что-то вычисляют чисто из данных стрима и шлют alert - полно, но чтобы alert посылался в результате такого комбинированного анализа - нету. Конечно имеется в виду almost real time solution=alerts. Основная мысль - warehouse data, нужные для принятия решения кешировать в память ( Redis) , делать из этого дела dataframe какой нибудь и джойнить его с данными в стриме. AWS KInesis предлагает нечто подобное в Kinesis Analytics. Называется конечно по другому, но думаю делает примерно то же самое с dataframes
этот вон пожалуйста сидит ждёт в облаке. Чего бы сразу туда не поставить все чем мордоваться и лепить какой нибудь процессинг на Postgres чтобы завтра все переделывать?
