Django: Sessions not working as expected on Heroku

Users keep getting logged out and sessions are not persisting on my Django app on Heroku. Users can log in, but they will be randomly logged out—even on the /admin/ site.

Is there anything I'm doing wrong with my Django/Heroku config?

Currently running Django 1.11.16 on Standard Dynos.

settings.py

SECRET_KEY = os.environ.get("SECRET_KEY", "".join(random.choice(string.printable) for i in range(40)))  SESSION_COOKIE_DOMAIN = ".appname.com" CSRF_COOKIE_DOMAIN = ".appname.com"  SECURE_SSL_REDIRECT = True  # ...  MIDDLEWARE_CLASSES = [     'django.middleware.security.SecurityMiddleware',     'django.contrib.sessions.middleware.SessionMiddleware',     'django.middleware.common.CommonMiddleware',     'django.middleware.csrf.CsrfViewMiddleware',     'django.contrib.auth.middleware.AuthenticationMiddleware',     'django.contrib.auth.middleware.SessionAuthenticationMiddleware',     'django.contrib.messages.middleware.MessageMiddleware',     'django.middleware.clickjacking.XFrameOptionsMiddleware', ]   TEMPLATES = [     {         'BACKEND': 'django.template.backends.django.DjangoTemplates',         'DIRS': [os.path.join(BASE_DIR, 'templates/')],         'APP_DIRS': True,         'OPTIONS': {             'context_processors': [                 'django.template.context_processors.debug',                 'django.template.context_processors.request',                 'django.template.context_processors.csrf',                 'django.contrib.auth.context_processors.auth',                 'django.contrib.messages.context_processors.messages',             ],         },     }, ]  # ...  DATABASES = {     'default': {         'ENGINE': 'django.db.backends.postgresql_psycopg2',         'NAME': 'appname',     } }  # https://devcenter.heroku.com/articles/python-concurrency-and-database-connections db_from_env = dj_database_url.config(conn_max_age=500) DATABASES['default'].update(db_from_env) 

1 Answers

Answers 1

The problem was that SECRET_KEY was not static on Heroku. The SECRET_KEY changing was breaking sessions. The fix is to add a static SECRET_KEY to Heroku config:

heroku config:set SECRET_KEY=`openssl rand -base64 32` 

Read More

hibernate second level cache with Redis -will it improve performance?

I am currently developing application using Spring MVC4 and hibernate 4 .I have implemented hibernate second level cache for performance improvement .If I use Redis which is in-memory data structure store, used as a database, cache etc, performance will increase but will it be a drastic change?

4 Answers

Answers 1

Drastic differences you may expect if you cache what is good to be cached and avoid caching data that should not be cached at all. Like beauty is in the eye of the beholder same is with the performance. Here are several aspects you should have in mind when using hibernate AS second level cache provider:

No Custom serialization - Memory intensive
If you use second level caching you would not be able to use fast serialization frameworks as Kryo and will have to stick to java serializeable which sucks.

On top of this for each entity type you will have a separate region and within each region you will have entry for each key of each entity. In terms of memory efficiency this is inefficient.

Lacks ability to store and distribute rich objects
Most of the modern caches also present computing grid functionality having your objects fragmented into many small pieces decrease your ability ability to execute distributed tasks with guaranteed data co-location. That depends a little bit on the Grid provider, but for many would be limitation.

Sub optimal performance
Depending on how much performance you need and what type of application you are having using hibernate second level cache might be a good or a bad choice. Good in terms that it is plug and play...."kind of..." bad because you will never squeeze the performance you would have gained. Also designing rich models mean more upfront work and more OOP.

Limited querying capabilities ON the Cache itself
That depends on the cache provider , but some of the provider really are not good doing JOINs with Where clause different than the ID. If you try to build and in memory index for a query on Hazelcast for example you will see what I mean.

Answers 2

Yes, if you use Redis, it will improve your performance.

No, it will not be a drastic change. :)

https://memorynotfound.com/spring-redis-application-configuration-example/

http://www.baeldung.com/spring-data-redis-tutorial

the above links will help you to find out the way of integration redis with your project.

Answers 3

Your question was already discussed here. Check this link: Application cache v.s. hibernate second level cache, which to use?

This was the most accepted answer, which I agree with:

It really depends on your application querying model and the traffic demands.

1. Using Redis/Hazelcast may yield the best performance since there won't be any round-trip to DB anymore, but you end up having a normalized data in DB and denormalized copy in you cache which will put pressure on your cache update policies. So you gain the best performance at the cost of implementing the cache update whenever the persisted data changes.
2. Using 2nd level cache is easier to setup but it only stores entities by id. There is also a query cache, storing ids returned by a given query. So the 2nd level cache is a two step process that you need to fine tune to get the best performance. When you execute projection queries the 2nd level object cache won't help you, since it only operates on entity load. The main advantage of 2nd level cache is that it's easier to keep it in sync whenever data changes, especially if all your data is persisted by hibernate.

So, if you need ultimate performance and you don't mind implementing your cache update logic that ensures a minimum eventual consistency window, then go with an external cache.

If you only need to cache entities (that usually don't change that frequently) and you mostly access those through Hibernate entity loading, then 2nd level cache can help you.

Hope it helps!

Answers 4

It depends on the movement.

If You have 1000 or more requests per second and You are low on RAM, then Yes, use redis nodes on other machine to take some usage. It will greatly improve your RAM and request speed.

But If it's otherwise then do not use it.

Remember that You can use this approach later when You will see what is the RAM and database Connection Pool usage.

Read More

Downside of many caches in spring

Due to the limitation of not being able to evict entries based on a partial key, I am thinking of a workaround using the cache name as my partial key and evicting all (there would only be one) entries in the cache. For example, let's say there are 2 key-value pairs like so:

"123@name1" -> value1, "124@name2" -> value2

Ideally, at the time of eviction, I would like to remove all keys that contain the string "123". However, as this is not supported, the workaround I'm thinking of is to have the following:

"123" cache: "name1" -> value1

"124" cache: "name2" -> value2

Then at eviction, I would simply specify to remove all keys in "123" cache

The downside of this of course is that there would be a lot of different caches. Is there any performance penalty to this?

From reading this, it seems Redis at least only uses the cache name as a prefix. So it is not creating multiple separate caches underneath it. But I would like to verify my understanding.

I am also looking to use Redis as my underlying cache provider if that helps.

1 Answers

Answers 1

You can use few approaches to overcome this :

1. Use grouped data structures like sets, sorted sets and hashes : Each one of them supports really high number of member elements. So you can use them to store your cache items,and do the relevant lookups. However, do go through the performance difference ( would be very small ) on this kind of lookup compared to a direct key-value lookup. Once you want to evict a group of cache keys which are of similar type, you just remove that data structure key from redis.

2. Use redis database numbers : You would need to edit redis.conf to increase maximum number of redis database numbers possible. Redis databases are just numbers which provide namespacing in which your key-values can lie. To group similar items, you would put them in the same database number, and empty the database with a single command whenever you want to flush that group of keys. The caveat here is that, though you would be able to use same redis connection, you would have to switch databases through redis SELECT command

Read More

Automatically reconnect Storm Topology to Redis Cluster on Redis restart

I have created a Storm topology which connects to Redis-cluster using Jedis library. Storm component always expects that Redis is up and running and only then it connects to Redis and subscribes the events.Currently we use pub-sub strategy of Redis.

Below is the code sample that explains my Jedis Connectivity inside Storm to for Redis.

try {     jedis.psubscribe(listener, pattern); } catch(Exception ex) {     //catch statement here. } finally {     pool.returnResource(jedis); }  ....  pool = new JedisPool(new JedisPoolConfig(), host, port); //redis host port  ListenerThread listener = new ListenerThread(queue, pool, pattern); listener.start(); 

EXPECTED BEHAVIOUR

Once Redis dies and comes back online, Storm is expected to identify the status of Redis. It must not need a restart in case when Redis die and come online.

ACTUAL BEHAVIOUR

Once Redis restarts due to any reason, I always have to restart the Storm topology as well and only then it starts listening back to Redis.

QUESTION

How can I make Storm listen and reconnect to Redis again after Redis is restarted? any guidance would be appreciated, viz. docs, forum answer.

1 Answers

Answers 1

This is a common issue with apache-storm where connection thread is alivein stale condition, although the source from where you are consuming is down/restarted. Ideally it should retry to create new connection thread instead reusing the existing one. Hence the Idea is to have it automated it by by detecting the Exception (e.g. JMSConnectionError in case of JMS).

refer this Failover Consumer Example which will give you brief idea what to do in such cases.(P.S this is JMS which would be JMS in redis your case.)

The Steps would be something like this.

1. Catch Exception in case of ERROR or connection lost.
2. Init connection (if not voluntarily closed by program) from catch.
3. If Exception got to step 1.

Read More

RedisCacheManager Not Updating keyspace_misses

I’m using the spring-boot spring-data-redis 1.8.9.RELEASE RedisCacheManager implementation of CacheManager for caching. One metric that I want visibility into is the cache hit/miss ratio. To get that, I’m extracting the keyspace_hits and keyspace_misses exposed via the redis server which can also be viewed via the redis_cli with INFO STATS. The problem is that RedisCacheManager never registers cache misses, i.e. keyspace_misses never increments even if there is a cache "miss".

Debugging the code, I see that spring-data-redis actually checks to see if the key EXISTS in redis before retrieving it. I see the sense with this approach however when EXISTS is executed against the redis server, it does not register a cache miss.

Is there any way to use RedisCacheManager and register cache misses? I know I can use other redis objects to accomplish this but I was wondering if it could be done with the standard CacheManager implementation?

Edit

The ideal solution won't add a great deal of overhead and I am unable to edit the configuration of the redis server.

Code that RedisCacheManager uses when retrieving an element from cache. Notice Boolean exists:

public RedisCacheElement get(final RedisCacheKey cacheKey) {     Assert.notNull(cacheKey, "CacheKey must not be null!");     Boolean exists = (Boolean)this.redisOperations.execute(new RedisCallback<Boolean>() {         public Boolean doInRedis(RedisConnection connection) throws DataAccessException {             return connection.exists(cacheKey.getKeyBytes());         }     });     return !exists ? null : new RedisCacheElement(cacheKey, this.fromStoreValue(this.lookup(cacheKey))); } 

The above code will execute these commands on redis viewable via MONITOR on a cache miss. Notice again that EXISTS is executed as per the code:

After the above commands are executed, keyspace_misses is not incremented even though there was a cache miss:

0 Answers

Read More

Microsoft.Extensions.Caching.Redis select different database than db0

a question on understanding which redis database is used and how it can be configured.

i have a default ASP.NET Core Web Application and a default configured local redis-server (containing 15 databases)

Over Package Management Console i have installed:

Install-Package Microsoft.Extensions.Caching.Redis 

Redis is configured in Startup.cs like this:

public void ConfigureServices(IServiceCollection services) {     services.AddMvc();      services.AddDistributedRedisCache(option =>     {         option.Configuration = "127.0.0.1";         option.InstanceName = "master";     }); } 

The code to read and write values into the cache is taken from the docs:

var cacheKey = "TheTime"; var existingTime = _distributedCache.GetString(cacheKey); if (!string.IsNullOrEmpty(existingTime)) {     return "Fetched from cache : " + existingTime; } else {     existingTime = DateTime.UtcNow.ToString();     _distributedCache.SetString(cacheKey, existingTime);     return "Added to cache : " + existingTime; } 

But this code only uses the default database db0 no matter what i configure.

E.g. using this configuration:

services.AddDistributedRedisCache(option => {     option.Configuration = "127.0.0.1";     option.InstanceName = "db6"; }); 

leads to:

What do i have to configure to use e.g. db6?

Do i have to use Stackexchange.Redis for this?

1 Answers

Answers 1

Microsoft.Extensions.Caching.Redis is using Stackexchange.Redis to connect to Redis.

The Configuration string is documented on StackExchange.Redis. That said, you should be able to do:

services.AddDistributedRedisCache(option => {     option.Configuration = "127.0.0.1;defaultDatabase=4";     option.InstanceName = "master"; }); 

Read More

Namespace for redis-queue queue in django app

I have a django app that uses redis-queue for managing long running tasks in the background. I have it set up and running (if in a stupid configuration), but I'm unclear what the appropriate namespace to store my queue is.

I set up the rq worker as recommended in the docs:

#rqsetup.py import os import redis from rq import Worker, Queue, Connection   listen = ['high', 'default', 'low'] redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379') conn = redis.from_url(redis_url)  if __name__ == '__main__':     with Connection(conn):         worker = Worker(map(Queue, listen))         worker.work() 

I can add the queue to my view:

# views.py from rqsetup import conn from rq import Queue from somewhere import bgtask  def myview(request):     q = Queue(connection=conn)     job = q.enqueue(bgtask)     return render(request, 'somepage.html') 

This is obviously dumb, because then the queue and the job are lost after the view is returned. The whole point is to have the queue in some namespace where I can access it again later.

However, I can't figure anywhere else to put it. If I try to in rqsetup, I just get import errors, or it imports out of order before its set up properly. I don't really have a good guess of where else it should go.

I otherwise have a normally structured app:

myproject  -myproject   -__init__.py   -settings.py   -urls.py   -wsgi.py  -myapp   -apps.py   -somewhere.py   -views.py  -manage.py  -Procfile  -requirements.txt  -rqsetup.py 

Where should a redis-queue queue be stored in a django project?

1 Answers

Answers 1

Turns out django_rq provides the queue namespace for you! settings.py should have something like this:

RQ_QUEUES = {     'default': {         'HOST': 'localhost',         'PORT': 6379,         'DB': 0,         'PASSWORD': 'some-password',         'DEFAULT_TIMEOUT': 360,     },     'high': {         'URL': os.getenv('REDISTOGO_URL', 'redis://localhost:6379/0'), # If you're on Heroku         'DEFAULT_TIMEOUT': 500,     },     'low': {         'HOST': 'localhost',         'PORT': 6379,         'DB': 0,     } } 

Then you just just get the queue from the django_rq namespace: queue = django_rq.get_queue('high').

Read More

Redis – Failed opening .rdb for saving: Permission denied

I am using redis version 3.0.6. The redis-server process is being run by the redis user.

Suddenly from 5 days after 24 hours redis began failing "opening .rdb for saving." It was working properly before this.

As you can see in the snippet from the logs below, Redis was behaving normally, and then started failing. Power-cycling the server later resolved the issue.

1427:M 24 May 01:09:05.102 * Background saving started by pid 2493     2493:C 24 May 01:09:34.916 * DB saved on disk 2493:C 24 May 01:09:34.917 * RDB: 310 MB of memory used by copy-on-write 1427:M 24 May 01:09:34.950 * Background saving terminated with success 1427:M 24 May 01:14:35.026 * 10 changes in 300 seconds. Saving... 1427:M 24 May 01:14:35.036 * Background saving started by pid 2494 2494:C 24 May 01:15:04.329 * DB saved on disk 2494:C 24 May 01:15:04.330 * RDB: 298 MB of memory used by copy-on-write 1427:M 24 May 01:15:04.408 * Background saving terminated with success 1427:M 24 May 01:20:05.008 * 10 changes in 300 seconds. Saving... 1427:M 24 May 01:20:05.018 * Background saving started by pid 2499 2499:C 24 May 01:20:33.830 * DB saved on disk 2499:C 24 May 01:20:33.831 * RDB: 330 MB of memory used by copy-on-write 1427:M 24 May 01:20:33.843 * Background saving terminated with success 1427:M 24 May 01:23:46.966 # Failed opening .rdb for saving: Read-only file system 1427:M 24 May 01:25:34.029 * 10 changes in 300 seconds. Saving... 1427:M 24 May 01:25:34.038 * Background saving started by pid 2500 2500:C 24 May 01:25:34.038 # Failed opening .rdb for saving: Read-only file system 1427:M 24 May 01:25:34.139 # Background saving error 1427:M 24 May 01:25:40.059 * 10 changes in 300 seconds. Saving... 1427:M 24 May 01:25:40.064 * Background saving started by pid 2501 2501:C 24 May 01:25:40.064 # Failed opening .rdb for saving: Read-only file system 1427:M 24 May 01:25:40.165 # Background saving error 1427:M 24 May 01:25:46.080 * 10 changes in 300 seconds. Saving... 1427:M 24 May 01:25:46.085 * Background saving started by pid 2502 2502:C 24 May 01:25:46.085 # Failed opening .rdb for saving: Read-only file system 1427:M 24 May 01:25:46.186 # Background saving error 1427:M 24 May 01:25:52.100 * 10 changes in 300 seconds. Saving... 1427:M 24 May 01:25:52.105 * Background saving started by pid 2503 2503:C 24 May 01:25:52.105 # Failed opening .rdb for saving: Read-only file system 1427:M 24 May 01:25:52.206 # Background saving error 

So, my question: how could this happen? Please give me proper solution for this.

1 Answers

Answers 1

The "Read-only file system" I think is the key here. It's possible the device it's trying to write to is mounted incorrectly but since it happened randomly, the system may have forced the filesystem into readonly mode. There's a number of conditions that can trigger the operating system to put the filesystem into a read-only mode. This can mean that the filesystem became corrupt or there was some other filesystem consistency issue. If you're hosting on a cloud provider and the disk is network-backed like EBS in AWS, this can be triggered by a temporary network issue. Sometimes the issues are momentary and either force remounting the partition (or power cycling the server) will fix the issue. Other times it's permanent, but since your server came back up just fine, that would appear to not be the case. But the true fix for this would lie in your hardware setup which wasn't detailed.

This answer is related albeit thin on the "why": Failed opening the RDB file ... Read-only file system

Read More

Redis::TimeoutError in Rails application

I keep getting Redis::Timeout error in my application (both in UI and in background jobs). I am using AWS ElastiCache service for Redis.

This is how I create a Redis connection. In my config/application.rb , I have:

$redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT, db: REDIS_DB) 

How can I avoid getting timeout errors? I am using the default connection settings as follows:

> $redis.client.options[:reconnect_attempts]  => 1  > $redis.client.options[:timeout]  => 5.0  > $redis.client.options[:tcp_keepalive]  => 0  > $redis.client.options[:inherit_socket]  => false 

1 Answers

Answers 1

You should pool your Redis connections with the help of the Connection Pool Gem and increase the timeout value if the problem persists:

ConnectionPool.new(size: 5, timeout: 3) {Redis.new({:host => 'localhost', :port => 6379, :db => 1, :timeout => 240})} 

Redis Gem

Read More

Which nosql option relative to stored procedures and large arrays?

I have a use case for a nosql data store but I don't know which one to use:

Each document in my data store has a key for _id and another key as an array of objects. Each object hash element of this array has a key for _elementid and another for color.

I want my server proxy to send an update request to the data store with a substring used as regex that qualifies all documents whose _id matches the regex. I then want to push an element onto the array of each document of this output. This new element will have the same color for each unshift but the _elementid will be unique for each.

Is there a nosql option out there that offers this kind of stored procedure? Does it have limits on the length of the array?

* EDIT *

(1) DOCUMENT A:

{     _id : "this_is-an-example_10982029822",     dataList : [         {             _elementid : "999999283902830",             color : "blue",          }, {             _elementid : "99999273682763",             color : "red"         }     ] }  DOCUMENT B:   {     _id : "this_is-an-example_209382093820",     dataList : [         {             _elementid : "99999182681762",             color : "yellow"         }     ] } 

(2) EXAMPLE OF UPDATE REQUEST

(let [regex_ready_array   ["this_is-an-example" "fetcher" "finder"]       fetch_query_regex   (str "^" (clojure.string/join "|^" regex_ready_array))       element_template    {                                 :_elementid { (rand-int 1000000000000000) }                                 :color      "green"                           }       updated_sister_objs (mc/bulk-update connection "arrayStore" {:_id {$regex fetch_query_regex }} "unshift" element_template)]) 

(3) DOCUMENT A:

{     _id : "this_is-an-example_10982029822",     dataList : [         {             _elementid : "999999146514612",             color : "green",          }, {             _elementid : "999999283902830",             color : "blue",          }, {             _elementid : "99999273682763",             color : "red"         }     ] }  DOCUMENT B:   {     _id : "this_is-an-example_209382093820",     dataList : [         {             _elementid : "9999997298729873",             color : "green",          }, {             _elementid : "9999918262881762",             color : "yellow"         }     ] } 

* EDIT 2 *

(1) the dataList array could be large (large enough that MongoDB's 16mb document size limit would present an issue);

(2) the _elementid values to be assigned to the additional dataList elements will be different for each new element and the the store will auto assign these as random number values

(3) a single update request should apply all updates, rather than one update per additional element;

(4) the OP is looking for a compare-and-contrast between several 'nosql solutions' which MongoDB, Cassandra, Redis and CouchDB being suggested as possible candidates.

1 Answers

Answers 1

By Seeing your question. I understand you are using JSONs and Clojure.

Lets see which are good NoSQL for JSONs. Quick overview of populor NoSQL

1. Apache Cassandra : Data Model in Cassandra is essentially a hybrid between a key-value and a column-oriented (or tabular) database management system. Its data model is a partitioned row store with consistency.

2. Redis: Redis maps keys to types of values.It has some abstract datatypes other than string like List, Sets, Sorted Sets, Hash Tables, Geospatial data.

3. Apache CouchDB : CouchDB manages a collection of JSON documents.

4. MongoDB : CouchDB manages a collection of BSON documents. BSON is Binary JSON http://bsonspec.org/spec.html.

If you are using lots of JSON payload you could use MongoDB or Apache CouchDB. But you want to update JSONs based on REGEX.

Lets check REGEX capability of CouchDB and MongoDB

• It can be done easily with MAP Reduce in Both CouchDB and MongoDB

  Regex Select: db.student.find( { f_name: { $regex: 'this_is-an-example.*'} } ).pretty();

• MongoDB: In mongodb we have regex operations. I have tried it and it works fine.

Reference

1. https://docs.mongodb.com/manual/reference/operator/query/regex/

2. mongoDB update statement using regex

3. https://www.w3resource.com/mongodb/mongodb-regex-operators.php

   • CouchDB: I haven't tried CouchDB with Regex but as far I know it is possible. Regex function is available as per CouchDB documentation.

   { "selector": { "afieldname": {"$regex": "^A"} } }

Reference

1. http://docs.couchdb.org/en/2.0.0/api/database/find.html
2. Temporary couchdb view of documents with doc_id matching regular expression

You could you either of this MongoDB and CouchDB. Lots of resources are avalible for MongoDB.

Read More

How to send data from client to redis and then after to laravel

I'm using laravel and redis for real time chat. I can fire event from my laravel and receiving that data to client side.

My problem is how can i send something from client and then receive it to redis and pass it to laravel

E.g How can i check if user has read the chat message.

Code :

var express = require('express'); var app     = express(); var server  = require('http').createServer(app); var io      = require( 'socket.io' ).listen( server ); var redis = require('redis'); var port = process.env.PORT || 8888; server.listen(port,'x.x.x.x');  io.on('connection', function (socket) {    console.log("Connected"); }); var redisClient = redis.createClient(); redisClient.psubscribe(['get_message','read_message']); redisClient.on("pmessage", function(channel, pattern, message) {     console.log(channel); // i can see get_message on this line in console but not read_message });  //Also tried this io.on('read_message', function (socket) {    console.log(socket); }); //Also this redisClient.on('read_message', function (socket) {    console.log(socket); });  redisClient.on('disconnect', function() {     console.log("Disconnected");     redisClient.quit(); }); 

Note : I'm emitting data from IOS app.

3 Answers

Answers 1

Redis is a key/value store engine. You should treat is like a database.

A client sends a request to a webserver that receives this request and decides what to do with it. For requests that require server-side processing, it will pass that request to a server-side "engine" for processing, in your case, probably PHP-FPM, which then passes it to a PHP daemon that will then execute the request.

Redis is not able to interpret a request in this manner. Therefore, you must intercept the request with Laravel and then send it to Redis. Not the other way around.

If you're trying to have Laravel get the information from Redis, you'll want to use Redis' pub/sub feature. Then you can have Laravel subscribe to Redis updates, get the updates and persist or handle the data however you want.

The phpredis lib supports the pub/sub functionality.

Here is an example of the PHP implementation with that lib. https://xmeng.wordpress.com/2011/11/14/pubsub-in-redis-using-php/

Answers 2

You can leverage Broadcasting in Laravel for the same. Also you can broadcast between multiple clients using whisper. Also there is a redis based broadcaster in it.

First you need to include its package in composer.

composer require pusher/pusher-php-server "~3.0" 

Configure the credentials in app/broadcasting.php

'options' => [ 'cluster' => 'eu', 'encrypted' => true ], 

You need to configure broadcasting queue as well.

Sending Broadcast

event(new ShippingStatusUpdated($update)); //Broadcasts to everyone, avail to working scope as well broadcast(new ShippingStatusUpdated($update));//Broadcasts only to others 

Receiving Broadcast

Instantiate socket ID for echo [included in header as X-Socket-ID]

var socketId = Echo.socketId(); 

Make sure you have echo installed

npm install --save laravel-echo pusher-js 

Create echo instance

import Echo from "laravel-echo"  window.Echo = new Echo({     broadcaster: 'pusher',     key: 'your-pusher-key' }); 

Listening events

Echo.channel('orders')     .listen('OrderShipped', (e) => {         console.log(e.order.name);     }); 

Broadcast an event to other connected clients without hitting your Laravel application at all

Echo.channel('chat')     .whisper('typing', {         name: this.user.name     }); 

Laravel Broadcast Documentation

Answers 3

I think this tutorial is what you want:

Laravel 5 and Socket.io Tutorial

it contains tutorial for client side and server side

and for ios

you could use: Socket.IO-Client-Swift

check this also: Socket.IO on iOS

Read More

Number of expiring keys listed by info command in redis not consistent with what I can see

When I run the info command in redis-cli against a redis 3.2.4 server, it shows me this for expires:

expires=223518

However, when I then run a keys * command and ask for the ttl for each key, and only print out keys with a ttl > 0, I only see a couple hundred.

I thought that the expires is a count of the number of expiring keys but I am not even within an order of magnitude of this number.

Can someone clarify exactly what expires is meant to convey? Does this include both to-be-expired and previously expired but not yet evicted keys?

---

Update:

Here is how I counted the number of keys expiring:

  task count_tmp_keys: :environment do     redis = Redis.new(timeout: 100)     keys = redis.keys '*'     ct_expiring = 0      keys.each do |k|       ttl = redis.ttl(k)       if ttl > 0         ct_expiring += 1         puts "Expiring: #{k}; ttl is #{ttl}; total: #{ct_expiring}"         STDOUT.flush       end     end      puts "Total expiring: #{ct_expiring}"     puts "Done at #{Time.now}"   end 

When I ran this script it shows I have a total expiring of 78

When I run info, it says db0:keys=10237963,expires=224098,avg_ttl=0

Because 224098 is so much larger than 78, I am very confused. Is there perhaps a better way for me to obtain a list of all 225k expiring keys?

Also, how is it that my average ttl is 0? Wouldn't you expect it to be nonzero?

---

UPDATE

I have new information and a simple, 100% repro of this situation locally!

To repro: setup two redis processes locally on your laptop. Make one a slave of the other. On the slave process, set the following:

config set slave-serve-stale-data yes config set slave-read-only no 

Now, connect to the slave (not the master) and run:

set foo 1 expire foo 10 

After 10 seconds, you will no longer be able to access foo, but info command will still show that you have 1 key expiring with an average ttl of 0.

Can someone explain this behavior?

2 Answers

Answers 1

The expires just returns the size of keys that will expire not the time.

The source code of 3.2.4

long long keys, vkeys;  keys = dictSize(server.db[j].dict); vkeys = dictSize(server.db[j].expires); if (keys || vkeys) {     info = sdscatprintf(info,         "db%d:keys=%lld,expires=%lld,avg_ttl=%lld\r\n",         j, keys, vkeys, server.db[j].avg_ttl); } 

It just calculate the size of server.db[j].expires. (note j is the database index).

Answers 2

expires contains existing keys with TTL which will expire, not including already expired keys. Example ( with omission of extra information from info command for brevity ):

127.0.0.1:6379> flushall OK 127.0.0.1:6379> SETEX mykey1 1000 "1" OK 127.0.0.1:6379> SETEX mykey2 1000 "2" OK 127.0.0.1:6379> SETEX mykey3 1000 "3" OK 127.0.0.1:6379> info # Keyspace db0:keys=3,expires=3,avg_ttl=992766 127.0.0.1:6379> SETEX mykey4 1 "4" OK 127.0.0.1:6379> SETEX mykey5 1 "5" OK 127.0.0.1:6379> info # Keyspace db0:keys=3,expires=3,avg_ttl=969898 127.0.0.1:6379> keys * 1) "mykey2" 2) "mykey3" 3) "mykey1" 127.0.0.1:6379>  

Can you share your script which counts the keys ?

Read More

Auto save server architecture

I want to save a lengthy form's inputs at the server. But I don't think making db calls on each auto-save action is the best approach to go for.

What would constitute as a good approach to solve this?

Another problem is that I have 3 app servers. So in memory cache wouldn't work.

I was thinking keeping the data in redis and updating it on every call and finally updating the db. But since I have 3 servers how do I make sure the calls are in queue?

Can anyone help with the architecture?

3 Answers

Answers 1

But I don't think making db calls on each auto-save action is the best approach to go for.

That's the real question, let's start with that. Why would you think that? You want auto-save, right? This is the only thing that saves user work.

All the other options you listed (memcached/redis, in-process caching) - not only do they not save user work, they're ticking time bombs. Think of all things that can fail there: redis dies, network is split, the whole data center is hit by lightning.

Why create all the complexity when you can just... save? You may find out that it's not that slow (if this was your concern).

Answers 2

This solution adopted industry wide successfully. Configure a 3 node redis cluster which take care of replication of data. Writes happen only to the master node.

redis (master) - app server 1, redis (slave1) - app server 2, redis (slave2) - app server 3

Adding a slave is straighforward using the slaveof :port command. Replication is done over the wire (not temporary disk storage) Link -https://redis.io/topics/replication

Answers 3

This is a very classical problem faced when scaling your architecture, but lets come to scaling later as essentially your initial app requires many calls to the data base level lets optimize that first.

Since you have not given any details to your iops I'll describe below the approaches to solving it, in increasing order of load, all approaches are of cascading nature, that is the last approach actually is built on all previous solutions :

The Direct Database Approach {db calls on each auto-save action}:

client -> application layer -> database (save data here directly)

Where basically on any data update we directly propagate it to the database level. The main block in this schema is the database Things to take care in this approach :

• For the most popularly used relational databases like Mysql :

  • An insert query takes less time than an update query, for a college project you could keep updating the row against the same primary key and it would work beautifully.

  • But in any mid sized application where a single form modify takes 30-40 requests the update query would block your db resource.

  • So keep a addendum kind of schema, like maybe maintain a secondary key like status that keeps track of till which level the user has filled the form but keep inserting data for each update. And always read as per the most recent status inserted.

  • For further optimization use of indexes such as foreign key constraints should be applied

  When this step fails, next step being the database itself, the next step to optimise on is, The type of data you're dealing with, you can choose a schema-less db like mongo, dynamodb etc for non transactional data

• Use a schema-less db can be very helpful for large amounts of non-transactional data as inherently they allow for the addendum approach to the same row of data.

  • For additional optimization use secondary indexes to query data faster.

Involving The Application Layer Approach {Application layer caching }:

client -> application layer (save some data here then propagate later) -> database (Finally save here)

• When the simple database is unable to serve and scale as required the easiest way is to shift some load on to your API server . As horizontal scaling of the same is much easier and cheaper.
• It is exactly as you understand the memory cache approach, though do not go about designing your own - It takes years of understanding large scale web infrastructure, then also people are not able to design an efficient cache on the application layer.
• use something like Express Session , I was using this for a web app having 10 ec2 nodejs instances, storing about 15mb of user data per session . It scales beautifully, maintains unique user session data across all servers - so on each auto update save data to user session - on form submit write to database from session. Can be easily scaled by adding more api servers, best use case with use it to save data on redis {Why re-invent the wheel ?}

Re-inventing the wheel : Custom level application layer caching + db caching + db optimize

client -> application layer (save some data here) ->{ Add your db cache}-> database (save data here finally)

• This is where we come to designing our own thing using redis/Dynamodb/mongo to act as a cache for your primary database. (Please note if using a non transactional db in the first place go for the addendum approach - this is purely much more suited to scale transactional databases by adding a wrapper of a non transactional database)

• Also express session works in the same way actually by caching the data on redis at the application layer, always try to lessen the number of calls to db as much as possible.

• So if you have a fully functioning application layer caching and an optimized db then go for this approach, as it needs experienced developers and is resource intensive and usually employed for very large scale applications like I had a layer of redis caching after the express session for an application that avgd iops of 300k reqs per second

• The approach here would be to save data on user session, apply a lazy write back to your cache. Maritain a cache ledger then write to your main database. For the queuing approach in such a large scale system I had written an entire separate microservice that worked in background to transport data from session to redis to mysql, for your concern of how to maintain a queue read more about priority queues and background workers . I used Kue is a priority job queue backed by redis, built for node.js.

• Maintaining parallel and sequential queues

• php client for KUE
• Background Services

Read More

Proxy TCP stream (MySQL and Redis) with Nginx

I read about Nginx Fabric Model and it brings my attention to reconfigure how an application communicates to MySQL and Redis. If local Nginx instance can proxy HTTP traffic efficiently and fast, now it can also proxy TCP without worrying of network, even using database slave as master in case of emergency and potentially encapsulate database sharding. All the benefits can simplify application configuration and its logic, network (congestion, latency, timeouts, retries) won't be a focus in features development anymore.

I use latest Docker and set of containers: Nginx, Redis, MySQL. I tried the following configuration:

user  nginx; worker_processes  1;  error_log  /var/log/nginx/error.log info; pid        /var/run/nginx.pid;   events {   worker_connections  1024; }  stream {   upstream redis {     # prefer first server but limit connections     server 172.17.0.8:6379 weight=2 max_conns=1;     server 172.17.0.3:6379;   }    upstream mysql {     # use second server in case of failure     server 172.17.0.4:3306;     server 172.17.0.5:3306 backup;   }    server {     listen 6379 so_keepalive=on;     proxy_pass redis;   }    server {     listen 3306 so_keepalive=on;     proxy_pass mysql;   } } 

I have some questions:

• logging - how can I know which endpoint is in use, how many times did Nginx retry a particular request?

• real-time stats - is it possible to get throughput for stream module?

• from database sharding perspective - is it possible to dispatch request to sharded database based on some logic apart from $remote_addr?

Last question is quite important, I found modules ngx_stream_map_module and ngx_stream_split_clients_module but $remote_addr is not suitable for sharding, can we intercept cookie from http section and reuse in stream section where we don't have any headers? Can we inject Lua code in the stream section? Is ngx_stream_ssl_preread_module a solution for this problem, how to make it work for connection without encryption?

0 Answers

Read More

Redis tries to connect to localhost on Heroku instead of REDIS_URL

I have a Rails app which uses Redis for background jobs. On Heroku I use the Heroku Redis add-on. When I deploy to Heroku, it gives me this error:

Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 

It seems it tries to connect to localhost. I have both a REDIS_URL and REDIS_PROVIDER environment variable on Heroku. And this is how my redis.rb looks like:

if Rails.env.production?   uri = URI.parse(ENV["REDIS_URL"]) else   uri = URI.parse("redis://localhost:6379") end Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password) 

And this is my Procfile:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb worker: env TERM_CHILD=1 bundle exec rake environment resque:work QUEUE=* COUNT=1 

Any ideas why it is not working? Even if I change redis.rb so it only has the REDIS_URL as url, it gives the same error.

Update: added error trace:

remote:        Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED) remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:345:in `rescue in establish_connection' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:331:in `establish_connection' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:351:in `block in time' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:350:in `time' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-namespace-1.5.3/lib/redis/namespace.rb:435:in `call_with_namespace' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-namespace-1.5.3/lib/redis/namespace.rb:321:in `method_missing' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque/data_store.rb:100:in `redis_time_available?' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque/data_store.rb:15:in `initialize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque.rb:125:in `new' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque.rb:125:in `redis=' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-web-0.0.9/config/initializers/resque_config.rb:4:in `<top (required)>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `block in load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:259:in `load_dependency' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:648:in `block in load_config_initializer' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/notifications.rb:166:in `instrument' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:647:in `load_config_initializer' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:611:in `each' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:611:in `block in <class:Engine>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:30:in `instance_exec' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:30:in `run' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:55:in `block in run_initializers' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:44:in `each' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:44:in `tsort_each_child' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:54:in `run_initializers' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:352:in `initialize!' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/config/environment.rb:5:in `<top (required)>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `block in require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:259:in `load_dependency' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:328:in `require_environment!' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:448:in `block in run_tasks_blocks' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/sprockets-rails-3.2.0/lib/sprockets/rails/task.rb:62:in `block (2 levels) in define' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>' remote:        Errno::ECONNREFUSED: Connection refused - connect(2) for 127.0.0.1:6379 remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:206:in `rescue in connect_addrinfo' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:198:in `connect_addrinfo' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:239:in `block in connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:237:in `each' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:237:in `each_with_index' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:237:in `connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:313:in `connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:351:in `block in time' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:350:in `time' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-namespace-1.5.3/lib/redis/namespace.rb:435:in `call_with_namespace' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-namespace-1.5.3/lib/redis/namespace.rb:321:in `method_missing' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque/data_store.rb:100:in `redis_time_available?' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque/data_store.rb:15:in `initialize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque.rb:125:in `new' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque.rb:125:in `redis=' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-web-0.0.9/config/initializers/resque_config.rb:4:in `<top (required)>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `block in load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:259:in `load_dependency' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:648:in `block in load_config_initializer' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/notifications.rb:166:in `instrument' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:647:in `load_config_initializer' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:611:in `each' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:611:in `block in <class:Engine>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:30:in `instance_exec' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:30:in `run' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:55:in `block in run_initializers' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:44:in `each' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:44:in `tsort_each_child' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:54:in `run_initializers' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:352:in `initialize!' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/config/environment.rb:5:in `<top (required)>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `block in require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:259:in `load_dependency' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:328:in `require_environment!' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:448:in `block in run_tasks_blocks' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/sprockets-rails-3.2.0/lib/sprockets/rails/task.rb:62:in `block (2 levels) in define' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>' remote:        IO::EINPROGRESSWaitWritable: Operation now in progress - connect(2) would block remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:199:in `connect_addrinfo' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:239:in `block in connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:237:in `each' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:237:in `each_with_index' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:237:in `connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/connection/ruby.rb:313:in `connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:336:in `establish_connection' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:101:in `block in connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:293:in `with_reconnect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:100:in `connect' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:364:in `ensure_connected' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:351:in `block in time' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-3.3.3/lib/redis.rb:350:in `time' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-namespace-1.5.3/lib/redis/namespace.rb:435:in `call_with_namespace' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/redis-namespace-1.5.3/lib/redis/namespace.rb:321:in `method_missing' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque/data_store.rb:100:in `redis_time_available?' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque/data_store.rb:15:in `initialize' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque.rb:125:in `new' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-1.27.2/lib/resque.rb:125:in `redis=' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-web-0.0.9/config/initializers/resque_config.rb:4:in `<top (required)>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `block in load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:259:in `load_dependency' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:287:in `load' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:648:in `block in load_config_initializer' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/notifications.rb:166:in `instrument' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:647:in `load_config_initializer' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:611:in `each' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/engine.rb:611:in `block in <class:Engine>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:30:in `instance_exec' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:30:in `run' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:55:in `block in run_initializers' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:44:in `each' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:44:in `tsort_each_child' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/initializable.rb:54:in `run_initializers' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:352:in `initialize!' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/config/environment.rb:5:in `<top (required)>' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `block in require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:259:in `load_dependency' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `require' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:328:in `require_environment!' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/railties-5.0.1/lib/rails/application.rb:448:in `block in run_tasks_blocks' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/sprockets-rails-3.2.0/lib/sprockets/rails/task.rb:62:in `block (2 levels) in define' remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>' remote:        Tasks: TOP => environment remote:        (See full trace by running task with --trace) remote:  ! remote:  !     Precompiling assets failed. remote:  !     Attempted to access a nonexistent database: remote:  !     https://devcenter.heroku.com/articles/pre-provision-database remote:  ! remote:  !     Push rejected, failed to compile Ruby app. remote:  remote:  !     Push failed remote: Verifying deploy.... remote:  remote: !   Push rejected to backbone-app-staging. remote:  To git@heroku.com:backbone-app-staging.git  ! [remote rejected] rails5 -> master (pre-receive hook declined) error: failed to push some refs to 'git@heroku.com:backbone-app-staging.git' 

4 Answers

Answers 1

I can think of two possibilities:

• Is something setting ENV['REDIS_URL'] before the initializer runs? For instance maybe you have a .env file checked into git that is overriding the Heroku variable?

• You say this code is from redis.rb. Do you have a config/initializers/resque.rb also? What about config/resque.yml? Either of those may be also trying to open a Redis connection. (If you could post the entire stack trace of your error, you could confirm this or rule it out.) Or are you using Redis for anything other than Resque?

You could also do some printf debugging and change your initializer to say:

if Rails.env.production?   puts "production: #{ENV['REDIS_URL']}"   uri = URI.parse(ENV["REDIS_URL"]) else   puts "not production"   uri = URI.parse("redis://localhost:6379") end 

That should help you clarify what's going on. (You may need to use Rails.logger.info instead of puts.)

EDIT: That stack trace is very helpful! Sure enough, your own initializer isn't in there at all, but there is other code trying to load its own redis connection, here:

remote:        /tmp/build_329306a238b046dda86a54d29db48f4c/vendor/bundle/ruby/2.4.0/gems/resque-web-0.0.9/config/initializers/resque_config.rb:4:in `<top (required)>' 

If you look up that gem, you can see it is doing this:

require 'resque'  config = ENV.fetch("RAILS_RESQUE_REDIS", "127.0.0.1:6379") Resque.redis = config 

So I think the answer is to set RAILS_RESQUE_REDIS in addition to REDIS_URL. You can use the Heroku config:set command to do that.

Answers 2

You forgot to pass the USERNAME to the connector, and your REDIS_URL can have been changed by the service provider...

I recommend use

$redis = if Rails.env.production? ? Redis.new(ENV["REDIS_URL"]) : Redis.new("redis://localhost:6379") 

It would be even better use the env variable your service provider set, like ENV['REDISCLOUD_URL'], because if they change the url for any motive, like when you update a plan or they need to change their infra... they could just set their variable and it would be transparent for you, without any interruption of service.

If you maintain yourself the REDIS_URL, and they change their url, your app will lost access to until you figure out the URL should change and set the update the REDIS_URL again

OBS

If the problem is when setting up the worker... the problem can be even that it is not starting with the correct environment, at least if you run it on Rails 5, you should config your resque Rakefile to

#If you're using Rails 5.x, include the following in lib/tasks/resque.rb:  require 'resque/tasks' task 'resque:setup' => :environment 

Answers 3

Try setting the REDIS_URL var in the ~/.bash_profile of the user that the application runs under.

export REDIS_URL="redis://prod-redis-url:6379" 

This should ensure that it is set before your app starts up.

Answers 4

Inside of your config/initializes/set_redis_url.rb

ENV['REDIS_URL'] = ENV["REDISCLOUD_URL"] if ENV["REDISCLOUD_URL"] 

In my past i have used action cable to work with redis and my app wouldn't work without these lines in config/environments/production.rb

config.action_cable.allowed_request_origins = ['https://your-app.herokuapp.com',                                            'http://your-app.herokuapp.com']  config.action_cable.url = "wss://sitepoint-custom-messaging.herokuapp.com/cable" 

Read More

Expose docker container port to other machine

I have installed Redis in Docker using below command

docker run -d -p 6379:6379 redis:3.0.1 docker run -d -p 6380:6379 redis:2.8.20 

Now I need to access this redis instance from another machine

public static ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(IPOFDOCKERINSTALLEDMACHINE:6379); 

My App is hosted in another machine in different server.

Wnen i am running the app, below is the exception

It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING

Is there anything need to be changed in docker or oracle virtual machine ?

4 Answers

Answers 1

Connecting multiple containers across the network on different servers is a perfect use case for docker swarm. You should try to create a overlay network and connect the running containers to the swarm and that network, as described here. Depending on your knowledge of the swarm ecosystem, yous could try different solutions.

Starting with docker 1.12, and if you want to manually manage the containers you could run

# retrieve the last swarm version $ docker pull swarm # running your swarm manager on your server $ docker swarm init --advertise-addr $(hostname -I | awk '{print $1}') # creating a cross server container network $ docker network create --driver overlay redisnet 

This command will output a slave command to use on your node. This command will allow you to join the swarm as "slave" server. If you want to launch services from that server, you should output the following command, which wil give you the manager token.

$ docker swarm join-token manager  To add a manager to this swarm, run the following command: docker swarm join \ --token SWMTKN-1-1ewyz5urm5ofu78vddmrixfaye5mx0cnuj0hwxdt7baywmppav-0p5n6b7hz170gb79uuvd2ipoy \ <IP_ADDRESS>:2377 

When your nodes are on the swarm you can launch your redis services with replicas

$ docker service create --network redisnet \                         --name redis --replicas 1 redis:3.0.1 $ docker service create --network redisnet \                         --name old_redis --replicas 1 redis:2.8.20 $ docker service create --network redisnet --name app <APP_IMAGE> 

Now all your containers can make http calls using the service name as hostname for the specific service. Basically if you only need to access your redis services from your application, this should do it.

You can also expose ports using the same docker option with -p but you have to discover which server runs your service. However, this require you to follow the other answwers to check if you have any port blocking on your VM.

Moreover, other solutions exist like Kubernetes or Mesos, but swarm is like the official way to go.

Answers 2

Given your run command without any ip restriction in:

$ docker run -d -p 6379:6379 redis:3.0.1 $ docker run -d -p 6380:6379 redis:2.8.20 

And the netstat output that shows it bound to localhost in the comment:

$ netstat -na | grep 6379 && netstat -na | grep 6380  TCP 127.0.0.1:6379 0.0.0.0:0 LISTENING 

There is a mismatch between the port you published to all interfaces and the port that is listening on localhost. There are a few possibilities:

1. Something else is listening on 127.0.0.1:6379. Run a sudo netstat -lntp | grep 6379 to find the process using that port.

2. Most likely the containers are not running. I say this since there wasn't anything on 6380. Check if your containers are running with docker ps -a. The ps output will include any port bindings if the container is running. If the containers exist, even if exited, you can check their logs with docker logs container_id to see if there are any errors.

3. Docker is not running on your local host. If echo $DOCKER_HOST is pointing to an ip or hostname, your client is sending the commands there. This would also apply if you're running your commands inside a VM and checking the results on your physical host.

4. Unlikely, but you could have changed the default ip on dockerd to 127.0.0.1 with the --ip flag. By default, published ports listen on all interfaces (0.0.0.0).

5. Your run command listed above may not be accurate. If you published the port with docker run -d -p 127.0.0.1:6379:6379 redis:3.0.1 that could explain the binding to 127.0.0.1. Removing the 127.0.0.1: from the command will default to binding to all interfaces.

Answers 3

Docker is binding your exposed ports to localhost. That is why you are finding issues. In order to make Docker unbind those exposed ports to localhost, you must run the containers via changing the command you use to launch the containers:

docker run -d -p 0.0.0.0:6380:6379 redis:2.8.20 docker run -d -p 0.0.0.0:6379:6379 redis:3.0.1 

The part that changed is that now you are specifying both a host and a port for the local part (and leaving the container part like it was; a single port).

I hope that helps you mate! And please, be careful with this. It's not like something you'd typically want to achieve. Make sure you are not leaving your redis service unauthenticated if you are using this for something even close to production.

Answers 4

Based on your error;

It was not possible to connect to the redis server(s); to create a >disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING

I think your other machine cannot access the machine with docker containers.

1. Try ping IPOFDOCKERMACHINE.
2. If step 1 is not working, that means you cannot access the machine with docker containers from your other machine. You need to fix that first.

3. If step 1 is successful try telnet command to make sure you can access the ports require. Please refer this article about telnet command. This article shows how to enable telnet in Windows OS. telnet IPOFDOCKERMACHINE 6380 telnet IPOFDOCKERMACHINE 6379

4. If telnet fails, that means you don't have access to ports you need. You need to fix this problem before moving forward.

5. If step 3 is successful you need to make sure your docker containers are launched as David Gonzalez has shown in his reply above.

Read More

How to deploy a Meteor app which uses Redis?

I stumbled upon this package recently redis-oplog which seems to be a very good package. Given that I do not have any experience with Redis, I did some search and found out some say that Redis is even better than Mongo Oplog therefore I want to give this package a try with my Meteor project. However I have some questions regarding deployment before I can try it:

• Do I need to have separate servers dedicated for running Redis?
• If I can not afford to have servers for Redis, is it ok to run Redis in the same server with the Meteor app?
• If the my Meteor app has many instances and so does Redis (not sure if Redis could have/need many instances?), how do I make them work all together?
• In case I manage to use Redis for production, what changes should I make to my Mongo servers? because Mongo Oplog is no longer in use at that point

2 Answers

Answers 1

• Yes you need to have separate instance for redis, you must setup a fault tolerant system using redis-sentinal, you can find different configurations and setups with pros and cons here https://redis.io/topics/sentinel
• If you cannot afford to have servers for Redis you can run it on server where you have installed your mongodb instance if you have one.Last option is to run it where your meteor instance is, redis-oplog will suerly be more efficient than mongodb-oplog as per the data available.
• There are samples for redis architecture in the link given above. Also connecting to redis is same as connecting to mongodb
• Mongo changes depend on your other usage for mongo.

Answers 2

Ansible is a nice deployment tool for all kinds of software. We have good experience with the Ansible role for Redis by David Wittman https://galaxy.ansible.com/DavidWittman/redis/ this can deploy a single instance (for development, on one box with all our components) or as a production cluster. Meteor can also be deployed by roles from Galaxy, but I did not use those.

Read More

Redis command to get all available keys on Redis Cluster?

I am using this

redisManager.redisClient.keys('*example*', function (err, keys) { }) 

But it only gives keys from only one of the redis cluster. How can I get keys from all cluster?

1 Answers

Answers 1

You can't get keys for all nodes using a single command. You have to get keys for all nodes and merge them. Reference - https://github.com/antirez/redis/issues/1962

You can do something like.

var redis = require('redis');  redisConfig = new Array(     {"port": 1234, "host": "192.168.1.2"},     {"port": 5678, "host": "192.168.1.3"} );  keys    = new Array(); allKeys = new Array();  for(i = 0; i < redisConfig.length; i++){     redisClient = redis.createClient(redisConfig[i].port, redisConfig[i].host);           keys[i] = redisClient.keys('*example*', function (err, keys) {       allkeys = [...new Set([...allKeys ,...keys[i]])];     }) } 

Read More

Redis cluster performance - high timeout rate on low load

See strange behavior of redis cluster, which works totally fine on big load and starts to run with 50% timeout rate and unstable response times on low load.

We have same patter each day on periods of low load.

Any ideas what could cause such a strange pattern? Maybe some maintenance work this RedisCluster starts to do on low load time? Like slots rebalancing. Please recommend any settings or aspects to check.

Versions: Redis 2.0.7, Jedis 2.8.1

Configuration: 3 physical nodes with 9 master processes and 18 slaves.

JedisCluster Timeout = 5ms.

Load is 100% writes with setex.

This graphs are for JedisCluster response times, not actual RedisCluster times. "Sets" line here is successful sets actually, not total count.

1 Answers

Answers 1

Finally I found that it looks like network issue.

redis08(10.201.12.214) ~ $ redis-benchmark -h 10.201.12.215 -p 9006 ====== PING_INLINE ======   100000 requests completed in 91.42 seconds   50 parallel clients   3 bytes payload   keep alive: 1  0.00% <= 11 milliseconds  redis09(10.201.12.215) ~ $ redis-benchmark -h 10.201.12.215 -p 9006 ====== PING_INLINE ======   100000 requests completed in 1.41 seconds   50 parallel clients   3 bytes payload   keep alive: 1  99.46% <= 1 milliseconds  redis08 ~ $ ping lga-redis09 PING redis09 (10.201.12.215) 56(84) bytes of data. 64 bytes from redis09 (10.201.12.215): icmp_seq=1 ttl=64 time=10.7 ms 

Looking at collectd's "if_octets" we have enormous network activity on network interfaces on this time of low write activity. Nighttime load is like 10x in comparison with daytime load.

And it is caused by redis nodes which start to actively exchange information on this low load period. Iptraf top connections output: While on daytime top in this iptraf report belongs fully to actual redis clients with good write load.

Will post updates on this answer.

Read More

Spring Session Data Redis - Get Valid Sessions, Current User from Redis Store

My question is, in distributed web application is it possible to get the valid sessions from Redis Store using RedisOperationSessionRepository. (I mean I don't want to write explicit code for putting it into Redis store and then later read it, I want to understand if framework or spring-data-redis library provides that).

I am aware that Spring Redis is able to restore sessions and server restarts also preserve the login if the session is still valid (as it is backed by Redis)

One of the functionality I am looking for is to get all the possible log in users currently in the application. I am aware of SessionRegistryImpl, and this method. but what I noticed that this method is not backed by Redis and after server restarts, log in users are not returned.

 public List<Object> getAllPrincipals() {     return new ArrayList<Object>(principals.keySet()); } 

One of the functionality I can try is from Spring Session 1.1.0, Spring session find by username.

1. http://docs.spring.io/spring-session/docs/1.1.0.M1/reference/html5/guides/findbyusername.html
2. https://spring.io/blog/2015/11/17/spring-session-1-1-0-m1-released

I tried and it indeed returns me valid session result, but the problem is I still need to know all the current valid user names that are using this application. (I don't know how to get them using Redis Store, again I can store in Redis and get them, but I want to know if there is better approach).

This is the piece of code, this is the way I can get current user from one of the many users that are currently using the system, if I know the session id.

    final Session session = redisOperationsSessionRepository.getSession(sessionid);      final Object obj = session.getAttribute(HttpSessionSecurityContextRepository.SPRING_SECURITY_CONTEXT_KEY);     if (obj instanceof SecurityContext) {         final SecurityContext context = (SecurityContext) obj;         final Authentication authentication = context.getAuthentication();         if (authentication != null) {             final Object principal = authentication.getPrincipal();             if (principal != null && principal instanceof CurrentUser) {                 return (CurrentUser) principal;             }         }     } 

Now I can use above logic to get all the current user, but again I should all the valid session ids, which I don't know how to get from Redis store.

New Update : https://github.com/spring-projects/spring-session/issues/255 Here in this link, I can probably get all the session ids and look for active sessions in RedisOperationSessionRepository, but might result in performance issues.

I am not sure if I made myself clear, but can't we tell something to Redis using spring session api, just give me all the valid sessions and their current user that are currently log in. (based on last accessed time or something like that).

Thank you

1 Answers

Answers 1

Redis is basically a key-value store and does not provide such querying features.

However, you should be able to list all session using a KEYS request, then filter them on a last-activity basis, but with drawbacks mentioned in the github issue you have mentioned.

You should probably consider logging users activities in a datastore that supports relational querying, keeping Redis as a fast session store.

Read More