Showing posts with label sockets. Show all posts
Showing posts with label sockets. Show all posts

Friday, October 5, 2018

When to handle Socket.io notifications?

Leave a Comment

I am developing an IOS social application that is written in SWIFT.

The backend is PHP, MySQL (for event handling), + a bit of NodeJS, Socket.io (for realtime chat and notifications)


I have made the chat successfully:

When the user sends a message the Socket.io server handles it the following way:

  • it inserts the datas to the database
  • if successful then emits the message to all the participant users

/ so for this the backend is only the Socket.io server, which handles the database aswell


Works fine.

But then there are events that are not meant to be real time, but still I want to send a notification to the given user with Socket.io

for example: if a post has been liked, then send a noti to the posts owner

I have already written the PHP files for saving the like in the database, but

How should I do the notification part, safe?


I have came up with 3 ideas:

  1. The app sends a web request to my PHP+MySQL backend, it handles the data there, then after returning back "success", the application (SWIFT) sends a notification to the post owner (via Socket.io XCode pod)
func likePost(postId : Int, completion: @escaping (ActionResult?)->()){          let connectUrl = URL(string: appSettings.url + "/src/main/like.php")         var request = URLRequest(url: connectUrl!)         request.httpMethod = "POST"         let postString = "userId=\(userId)&session=\(session)&pId=\(postId)"         request.httpBody = postString.data(using: String.Encoding.utf8)           let task = URLSession.shared.dataTask(with: request) {             (data: Data?, response: URLResponse?, error: Error?) in              if error != nil {                 return completion(ActionResult(type: 0, code: 0, title: "error", message: "something went wrong"))             }             do {                  let responseJson = try JSONSerialization.jsonObject(with: data!, options: [])                 if let responseArray = responseJson as? [String: Any] {                      let responseStatus = responseArray["status"] as? String                     let responseTitle = responseArray["title"] as? String                     let responseMessage = responseArray["message"] as? String                       if responseStatus != "1" {                         return completion(ActionResult(type: 0, code: 0, title: "error", message: "something went wrong"))                     }                      // SUCCESS, SEND NOTI WITH SOCKET.IO                      socket.emit("notification_likedPost", ["postId": postId)                      return completion(ActionResult(type: 1, title: "success", message: "yay"))                  }             } catch {                 return completion(ActionResult(type: 0, code: 0, title: "error", message: "something went wrong"))             }         }         task.resume()     } 
  1. same, but after returning back "success" from the PHP, itself (the PHP file) handles the Socket.IO notification emitting as well (I think this is not possible, I haven't found any PHP->Socket.io plugins..)

-

  1. The app does not send anything to my web PHP+MySQL file, instead it sends the whole "like" process to my NodeJs, Socket.IO server, it handles it there, saves it to the database, then emits the notifications (Just like the real time chat part, but this would be a lot work because I have already written all the other code in PHP files)

The first case is the most ideal for me, but I am scared that it would be hackable..

Because if I do it the first way, the backend NodeJs+Socket.io server won't check if the liking process was successful (because it was checked client-sided)

so it is likely that anyone could send fake "post like" notifications, like a billion times.


Then maybe the second option would be great as well, so that back-end handles both checking, and notification sending, but sadly there's no Socket.io plugin for PHP

3 Answers

Answers 1

It would be much more simpler to ...

Forget PHP, Go full Nodejs:

Express (you can also combine it with handlebars & i18n for multi-language purpose)

With express you can build a router for incoming requests (GET,PUT,POST,...)

This means that you can use it to render pages with server-side dynamic data

const express = require('express'); const exphbs = require('express-handlebars'); const app = express();  // Register Handlebars view engine app.engine('handlebars', exphbs()); // Use Handlebars view engine app.set('view engine', 'handlebars');  var visit_counter = 0;  app.get('/', (req, res) => {   var time_stamp = Date.now(); visit_counter++   res.render('index',{"timestamp":time_stamp,"visits":visit_counter}); });  app.listen(3000, () => {   console.log('Example app is running → PORT 3000'); }); 

The views/index.hbs file would look like this :

<!doctype html> <html lang="en"> <head>     <meta charset="UTF-8">     <title>Example App</title> </head> <body>   <p> Current Time : {{timestamp}} </p>  <p> Total Visits : {{visits}} </p>  </body> </html> 

This above part is an example of server-side data being rendered in the final html.


Socket.io (if you want more than 1 instance of the server running, no problem, lookup socket.io-redis)

You can combine express with socket.io in different ways, you could even use cookie-based authentication for your socket protocol. so when an event is coming in you could actually tell 100% if its a legit user and its user-id.


To prevent the spam of likes... you have to control them somehow. You should store the action of the like, so it cant be repeated more than once for the same post (so user-id & post-id seem to be the important variables here)



Here comes the update :

Since you made quite clear that you want a php & nodejs combo :

Redis is an in-memory data structure store which can be used as a database, a cache and a message broker.

PHPRedis @Github

Redis PubSub with PHP and Node.JS

A quick example of Node.js reading PHP session on Redis

Using Redis, you can easily listen to php events from your nodejs instance.

I suggest that you also think about the future scaling of your system and give a try at learning more nodejs to be able to move on from php.

Answers 2

I understand your concern as your whole project has more concentration of PHP code as compared to other frameworks/languages. In order to rectify your problem, here is the Socket.io implementation for PHP v5.3 and above https://github.com/walkor/phpsocket.io.

With the help of this, you can use socket.io library in your PHP code. Below you can see an example of using Socket.io library in PHP.

use Workerman\Worker; use PHPSocketIO\SocketIO;  // listen port 2020 for socket.io client $io = new SocketIO(2020); $io->on('connection', function($socket){     $socket->addedUser = false;     // when the client emits 'new message', this listens and executes     $socket->on('new message', function ($data)use($socket){         // we tell the client to execute 'new message'         $socket->broadcast->emit('new message', array(             'username'=> $socket->username,             'message'=> $data         ));     });     // when the client emits 'add user', this listens and executes     $socket->on('add user', function ($username) use($socket){         global $usernames, $numUsers;         // we store the username in the socket session for this client         $socket->username = $username;         // add the client's username to the global list         $usernames[$username] = $username;         ++$numUsers;         $socket->addedUser = true;         $socket->emit('login', array(              'numUsers' => $numUsers         ));         // echo globally (all clients) that a person has connected         $socket->broadcast->emit('user joined', array(             'username' => $socket->username,             'numUsers' => $numUsers         ));     });     // when the client emits 'typing', we broadcast it to others     $socket->on('typing', function () use($socket) {         $socket->broadcast->emit('typing', array(             'username' => $socket->username         ));     });     // when the client emits 'stop typing', we broadcast it to others     $socket->on('stop typing', function () use($socket) {         $socket->broadcast->emit('stop typing', array(             'username' => $socket->username         ));     });     // when the user disconnects.. perform this     $socket->on('disconnect', function () use($socket) {         global $usernames, $numUsers;         // remove the username from global usernames list         if($socket->addedUser) {             unset($usernames[$socket->username]);             --$numUsers;            // echo globally that this client has left            $socket->broadcast->emit('user left', array(                'username' => $socket->username,                'numUsers' => $numUsers             ));         }    }); });  Worker::runAll(); 

Answers 3

You can create multiple web sockets channel. In your case, you have added one using socket.io in NodeJS. You can add another channel through php way.

You can listen to that channel the same way your are listening from NodeJS.

Few handy links 1. http://php.net/manual/en/book.sockets.php 2. How to create websockets server in PHP

Read More

Sunday, September 2, 2018

Create messaging system in python using socket programming

Leave a Comment

I am new to socket programming. I wanted to create a simple messaging system between the server and the client ( chat ). I have included my code below. I am expecting it to work as similar as chat system but it doesn't work. If the message is sent it should receive and print it out but only after giving the input the received string is printed. I am expecting it should run parallelly (receive a message and send a message).

Server :

import socket import time import threading  def get(s):     tm = s.recv(1024)     print("\nReceived: ",tm.decode('ascii'))  def set_(s):     i=input("\nEnter : ")     s.send(i.encode('ascii'))   serversocket = socket.socket()  host = socket.gethostname()  port = 9981  serversocket.bind((host,port))  serversocket.listen(1)  clientsocket,addr = serversocket.accept()  while(1):     t1=threading.Thread( target = get ,  args = (clientsocket,) )     t1.start()     t2=threading.Thread( target = set_ ,  args = (clientsocket,) )     t2.start()     time.sleep(10) clientsocket.close() 

Client:

import socket import threading import time def get(s):     tm = s.recv(1024)     print("\nReceived: ",tm.decode('ascii'))      def set_(s):     i=input("\nEnter : ")     s.send(i.encode('ascii'))  s = socket.socket() host = socket.gethostname() port = 9981 s.connect((host,port))  while(1):     t1=threading.Thread( target = get ,  args = (s,) )     t2=threading.Thread( target = set_ , args = (s,) )     t1.start()     t2.start()     time.sleep(10) s.close() 

Output (At Client) :

Enter: hello ------------------------------>(1)  Received: hello --------------------------->(3) 

Output (At Server) :

Enter: hello ------------------------------>(2)  Received :  hello ------------------------->(4) 

Expected Output:

Output (At Client) :

Enter: hello ------------------------------>(1)  Received: hello --------------------------->(4) 

Output (At Server) :

Received :  hello ------------------------->(2)  Enter: hello ------------------------------>(3) 

The number represents the order of execution.

1 Answers

Answers 1

There is an issue with the threading logic of your program. You should move the while(True) loops to the thread workers, and only start your threads once. As it stands, your code can only send/receive one message every 10 seconds.

Server:

import socket import threading  def get(s):     while True:         tm = s.recv(1024)         print("\nReceived: ",tm.decode('ascii'))  def set_(s):     while True:         i=input("\nEnter : ")         s.send(i.encode('ascii'))  serversocket = socket.socket() host = socket.gethostname() port = 9981 serversocket.bind((host,port)) serversocket.listen(1) clientsocket,addr = serversocket.accept() t1=threading.Thread( target = get ,  args = (clientsocket,) ) t1.start() t2=threading.Thread( target = set_ ,  args = (clientsocket,) ) t2.start() 

Client:

import socket import threading  def get(s):     while True:         tm = s.recv(1024)         print("\nReceived: ",tm.decode('ascii'))  def set_(s):     while True:         i=input("\nEnter : ")         s.send(i.encode('ascii'))  s = socket.socket() host = socket.gethostname() port = 9981 s.connect((host,port)) t1=threading.Thread( target = get ,  args = (s,) ) t2=threading.Thread( target = set_ , args = (s,) ) t1.start() t2.start() 

You'll need to handle closing the sockets differently, and the enter/received prints get out of sync after the first message due to the multithreaded nature of the program, but the input is still waiting.

Read More

Tuesday, July 31, 2018

fsockopen(): unable to connect not work with PHP (Connection timed out)

Leave a Comment

I have this code:

$domain = 'massag.com';  $hosts = array(); $mxweights = array(); getmxrr($domain, $hosts, $mxweights);  var_dump($hosts); var_dump($mxweights);  $host = gethostbynamel($hosts[0])[0]; var_dump($host);  $f = @fsockopen($host, 25, $errno, $errstr, 10);  if(!$f) {     var_dump('NOT CONNECTED'); } 

It is not connected to smtp server but when I use command

smtp:217.196.209.9

on mxtoolbox.com it is connected.

Am I doing something wrong with PHP code? I already tried replace $host to smtp.massag.com but not helped.

2 Answers

Answers 1

Remove @ in your call to fsockopen(); so you can see any potential errors you have happening in your configuration.

This code seems to be working fine to me.

$domain = 'massag.com';  $hosts = array();  $mxweights = array();  getmxrr($domain, $hosts, $mxweights);  var_dump($hosts); var_dump($mxweights);  $host = gethostbynamel($hosts[0])[0];  var_dump($host);  $f = fsockopen($host, 25, $errno, $errstr, 10);  if(!$f) {     var_dump('NOT CONNECTED'); } 

example output

Something to consider is that the getmxrr() function was not available on Windows platforms before php version 5.3.0. If you're on Windows ensure you're at least on that version of php or later.

Answers 2

Using dig to query the IP provided or its reverse DNS it shows that there are no MX records so errors are expected.

dig -x 217.196.209.9 MX | grep 'IN.*MX' ;9.209.196.217.in-addr.arpa.    IN      MX  dig smtp.miramo.cz MX | grep 'IN.*MX' ;smtp.miramo.cz.                        IN      MX 

But returns results on massag.com

dig massag.com MX | grep 'IN.*MX' ;massag.com.                    IN      MX massag.com.             85375   IN      MX      20 miramo3.miramo.cz. massag.com.             85375   IN      MX      10 smtp.miramo.cz. 

Finally, adding some tests to avoid unnecessary errors and using working domains

<?php $domain = 'massag.com';  if(getmxrr($domain, $hosts, $mxweights)){     print_r($hosts);     print_r($mxweights);     if(count($hosts) > 0){         $host = gethostbynamel($hosts[0])[0];         print("Found host: " . $host . "\n");          $f = fsockopen($host, 25, $errno, $errstr, 10);          if(!$f){             var_dump('NOT CONNECTED');         }     }else{         print("no MX record found\n");     } } ?> 

Result using tutorialspoint.com as domain:

    Array (     [0] => ALT2.ASPMX.L.GOOGLE.com     [1] => ASPMX.L.GOOGLE.com     [2] => ALT1.ASPMX.L.GOOGLE.com     [3] => ALT4.ASPMX.L.GOOGLE.com     [4] => ALT3.ASPMX.L.GOOGLE.com ) Array (     [0] => 5     [1] => 1     [2] => 5     [3] => 10     [4] => 10 ) Found host: 74.125.128.26 

Using the domain provided by OP (massag.com)

    Array (     [0] => smtp.miramo.cz     [1] => miramo3.miramo.cz ) Array (     [0] => 10     [1] => 20 ) Found host: 217.196.209.9 
Read More

Saturday, May 5, 2018

Connect to a socket.io server from a Node.js server using the 'ws' package

Leave a Comment

I have a Node.js server which utilizes the popular ws package for using web sockets. I'd like to use this library to connect to an third party server which is running socket.io.

If I were to use socket.io on my server, the connection code would be something like this:

const socket = socketIo('https://api.example.com/1.0/scores') 

I've attempted to connect to the same service using the ws package, and modifying the url:

const wsClient = new WebSocket('wss://api.example.com/1.0/scores'); 

but this results in the following:

Error: Unexpected server response: 200

Question: What needs to be done to connect to a third party server running socket.io from a server running the ws package?

Additional Info:

  • I've noticed in my searches that some people have suggested appending /socket.io/?EIO=3&transport=websocket to the end of the url. This does not throw the same error as above (> Error: Unexpected server response: 200) nor throw any visible error, but does not appear to work (no data is received from the remote server).
  • Using new WebSocket('ws://api.example.com/1.0/scores?EIO=3&transport=websocket'); to open the connection (via ws) results in the following stack trace:

    { Error: Parse Error     at Socket.socketOnData      at emitOne      at Socket.emit      // ...  } 

2 Answers

Answers 1

Because Socket.IO doesn't guarantee that there will be a WebSockets server hosted like you're seeming to expect, you should instead use their standard client package.

npm i socket.io-client 

Then use the package in your code:

const ioClient = require('socket.io-client')('https://example.com/1.0/scores') 

The full docs for socket.io-client are available on their GitHub repo.

Note: Honestly, though, it's just better at this point to use WebSockets instead if possible. WebSockets has become very well supported in browsers and is quite standard. Socket.IO is rarely necessary and is just bulky.

Answers 2

The socket.io api utilizes websockets but it also has a lot of other functions built on top of it in order to do things such as HTTP handshakes, session ids, and it can even handle fail overs to other protocols when needed.

You got half of the issue so far. Adding the line socket.io/?EIO=3&transport=websocket you're specifying parameters for the socket.io server to take.

EIO=3 specifies the version number for engine.io in which socket.io is using. In this case you are saying engine.io version = 3

transport=websocket specifies which transport protocol to use. As i said earlier, socket.io uses other protocols in cases such as fail overs. This portion forces socket.io to use websocket as the preferred protocol.

Now the next half is the WebSocket api itself. WebSocket allows for Extensions which includes different kinds of compression that are commonly used when sending data. Which I believe is what is causing your Parse Error

Try this (found here):

const WebSocket = require('ws'); const ws = new WebSocket('ws://server/socket.io/?EIO=3&transport=websocket', {    perMessageDeflate: false }); 

By setting perMessageDeflate: false you are specifying "Do not compress data". Since as i said this is a WebSocket Extension there are different variations as well. Try these instead if it doesn't work

  • x-webkit-deflate-frame
  • perframe-deflate

As a disclaimer this information is from the research that I have done. Im not a "socket.io specialist" so if there's anything i got wrong or missed please tell me and i'll edit the post.

Read More

Monday, April 9, 2018

WebSocket connection to 'wss://' Error during WebSocket handshake: Unexpected response code: 400

Leave a Comment

I use socket.io in my node.js app that's running on express.

Everything words fine on the local version (localhost) however when I switch to my production server (which is served via https using a custom certificate), I get the following error in my browser console:

websocket.js:112 WebSocket connection to 'wss://infranodus.com/socket.io/?EIO=3&transport=websocket&sid=j_WBxkPY_RlpF9_ZAANP' failed: Error during WebSocket handshake: Unexpected response code: 400

I made a research (issue referenced here) and it turns out this happens because my app / hosting provider blocks connections like wss and my socket.io falls back on AJAX to make requests (which functions OK, but sometimes there are bugs).

So I wanted to ask you if I could do modifications to my app to get rid of this error?

Just FYI currently all requests to http://infranodus.com are forwarded (via static .htaccess) to https://infranodus.com and my app.js file (the part of the server activation looks like that):

var http = require('http');  var app = express();  var server = http.Server(app); var io = require('socket.io')(server);  app.set('port', process.env.PORT || 3000); app.set('views', __dirname + '/views'); app.set('view engine', 'ejs'); 

and the way I require sockets in my front-end file:

<script src="/socket.io/socket.io.js"></script>

and then

var socket = io();

Maybe the problem is that I activate server in my node.js app using http and not https? But I would not like to switch to that because I don't know where my certificates are stored and I would not like to change the backend code too much.

Just in case, all the code for the app is available on https://github.com/noduslabs/infranodus

1 Answers

Answers 1

Use a secure URL for your initial connection, i.e. instead of "http://" use "https://". If the WebSocket transport is chosen, then Socket.IO should automatically use "wss://" (SSL) for the WebSocket connection too.

If you are not specifying any URL when you call io(), since it defaults trying to connect to the host that serves the page, either you have to provide url or change to https

  var socket = io.connect('https://localhost', {secure: true}); //remote url 
Read More

Monday, March 5, 2018

Alert(Level: Fatal, Description: Decode Error) - Forwarding Proxy

Leave a Comment

I'm trying to make a forwarding proxy but I keep getting an

Alert(Level: Fatal, Description: Decode Error) 

after the Client sends...

Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 

enter image description here

enter image description here

Any ideas as to what I'm doing wrong?

I can't seem to get a grasp on what the error even means. Does it mean the initial encrypted packet by the client fails to be decrypted by the server? If so, then why?

UPDATE 1

I just was looking at the packets and I noticed a significant difference between using my proxy, and not using the proxy.

The DFE key isn't being interpereted with my proxy.

enter image description here

enter image description here

1 Answers

Answers 1

Any ideas as to what I'm doing wrong?

You're not forwarding the exact amount of data that the proxy is supposed to forward.

But I see you're going further now than at the beginning of your question (good !)

You are implementing a proxy which forwards every single byte which it receives, in both ways, and either it sends too much to the server, or not enough. Check your code again for any conditions when you stop reading the input data to forward, be sure you're forwarding exactly everything. Nothing more, nothing less.

RFC 5246, about Decode Error :

decode_error A message could not be decoded because some field was out of the specified range or the length of the message was incorrect. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).

Read More

Friday, February 23, 2018

Arduino client hangs when trying to make a socket connection

Leave a Comment

I am working on a project that involves sockets in local network - I want to make a Java server (desktop application running on Windows) that will listen to and make connections with several clients - Arduino boards.

The problem is, code sticks while trying to make a connection. Here's the Java code:

monitorThread = new Thread(() -> {         try {             System.out.println("Creating socket...");             ServerSocket server = new ServerSocket(4444);             while (true) {                 System.out.println("Waiting for connection...");                 Socket client = server.accept();                 //NetworkManager.this.didConnect(client);                 System.out.println("Did establish connection");                 if (delegate != null) {                     delegate.didConnect(client);                 }             }         } catch (IOException exception) {             System.out.print(exception);         }     });     monitorThread.start(); 

and the Arduino code

#include <SPI.h> #include <Ethernet.h>  IPAddress serverIp(192, 168, 1, 101); int serverPort = 4444;  byte mac[] = {   0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xEF }; IPAddress ip(192, 168, 1, 178);  EthernetClient client;  void setup() {   Ethernet.begin(mac, ip);    Serial.begin(9600);   while (!Serial) {     ; // wait for serial port to connect. Needed for native USB port only   }    delay(1000);   // give the Ethernet shield a second to initialize:    Serial.println("connecting...");    if (client.connect(serverIp, serverPort)) {     Serial.println("connected.");   } else {     Serial.println("connection failed.");   } } 

What happens is both of those do not fail when trying to make a connection, but rather hang - Java server hangs on Socket client = server.accept();, but Arduino hangs as well - on client.connect(serverIp, serverPort)

Computer running Java server has a static IP (192.168.1.101).

I understand Java's server.accept() is a blocking call, so it will not proceed until a connection is made (what's why it runs in a separate thread), but what puzzles me is why Arduino hands as well.

Even if I try to connect to some other server IP and port - for example 64.233.187.99 (Google), it still hangs.

What am I doing wrong? Is there some additional setup that I haven't done? Can it have something to do with my network's settings?

1 Answers

Answers 1

Kindly try these steps and give it a try:

  • Turn off firewall on Windows (Java Server)
  • Check from another network connected pc that you can connect (telnet) to 192.168.1.101 on port 4444 (ie: telnet 192.168.1.101 4444)
  • Change delay(1000) line to delay(5000) to give arduino ethernet shield more time to initialize

Also please post your loop code. I am using exactly the same code to communicate arduino to a java socket server. Your code seems fine; it might be firewall or something. Make sure you check out telnet connection from another machine.

Read More

Monday, January 22, 2018

Separate computation from socket work in Python

Leave a Comment

I'm serializing column data and then sending it over a socket connection. Something like:

import array, struct, socket  ## Socket setup s = socket.create_connection((ip, addr))  ## Data container setup ordered_col_list = ('col1', 'col2') columns = dict.fromkeys(ordered_col_list)  for i in range(num_of_chunks):     ## Binarize data     columns['col1'] = array.array('i', range(10000))     columns['col2'] = array.array('f', [float(num) for num in range(10000)])     .     .     .      ## Send away     chunk = b''.join(columns[col_name] for col_name in ordered_col_list]     s.sendall(chunk)     s.recv(1000)      #get confirmation 

I wish to separate the computation from the sending, put them on separate threads or processes, so I can keep doing computations while data is sent away.

I've put the binarizing part as a generator function, then sent the generator to a separate thread, which then yielded binary chunks via a queue.

I collected the data from the main thread and sent it away. Something like:

import array, struct, socket from time import sleep try:     import  thread     from Queue import Queue except:     import _thread as thread     from queue import Queue   ## Socket and queue setup s = socket.create_connection((ip, addr)) chunk_queue = Queue()   def binarize(num_of_chunks):     ''' Generator function that yields chunks of binary data. In reality it wouldn't be the same data'''      ordered_col_list = ('col1', 'col2')     columns = dict.fromkeys(ordered_col_list)      for i in range(num_of_chunks):         columns['col1'] = array.array('i', range(10000)).tostring()         columns['col2'] = array.array('f', [float(num) for num in range(10000)]).tostring()         .         .          yield b''.join((columns[col_name] for col_name in ordered_col_list))   def chunk_yielder(queue):     ''' Generate binary chunks and put them on a queue. To be used from a thread '''      while True:            try:             data_gen = queue.get_nowait()         except:             sleep(0.1)             continue         else:                 for chunk in data_gen:                 queue.put(chunk)   ## Setup thread and data generator thread.start_new_thread(chunk_yielder, (chunk_queue,)) num_of_chunks = 100 data_gen = binarize(num_of_chunks) queue.put(data_gen)   ## Get data back and send away while True:    try:         binary_chunk = queue.get_nowait()     except:         sleep(0.1)         continue     else:             socket.sendall(binary_chunk)         socket.recv(1000) #Get confirmation 

However, I did not see and performance imporovement - it did not work faster.

I don't understand threads/processes too well, and my question is whether it is possible (at all and in Python) to gain from this type of separation, and what would be a good way to go about it, either with threads or processess (or any other way - async etc).

2 Answers

Answers 1

If you are trying to use concurrency to improve performance in CPython I would strongly recommend using multiprocessing library instead of multithreading. It is because of GIL (Global Interpreter Lock), which can have a huge impact on execution speed (in some cases, it may cause your code to run slower than single threaded version). Also, if you would like to learn more about this topic, I recommend reading this presentation by David Beazley. Multiprocessing bypasses this problem by spawning a new Python interpreter instance for each process, thus allowing you to take full advantage of multi core architecture.

Answers 2

You have two options for running things in parallel in Python, either use the multiprocessing (docs) library , or write the parallel code in cython and release the GIL. The latter is significantly more work and less applicable generally speaking.

Python threads are limited by the Global Interpreter Lock (GIL), I won't go into detail here as you will find more than enough information online on it. In short, the GIL, as the name suggests, is a global lock within the CPython interpreter that ensures multiple threads do not modify objects, that are within the confines of said interpreter, simultaneously. This is why, for instance, cython programs can run code in parallel because they can exist outside the GIL.


As to your code, one problem is that you're running both the number crunching (binarize) and the socket.send inside the GIL, this will run them strictly serially. The queue is also connected very strangely, and there is a NameError but let's leave those aside.

With the caveats already pointed out by Jeremy Friesner in mind, I suggest you re-structure the code in the following manner: you have two processes (not threads) one for binarising the data and the other for sending data. In addition to those, there is also the parent process that started both children, and a queue connecting child 1 to child 2.

  • Subprocess-1 does number crunching and produces crunched data into a queue
  • Subprocess-2 consumes data from a queue and does socket.send

in code the setup would look something like

from multiprocessing import Process, Queue  work_queue = Queue() p1 = Process(target=binarize, args=(100, work_queue)) p2 = Process(target=send_data, args=(ip, port, work_queue)) p1.start() p2.start() p1.join() p2.join() 

binarize can remain as it is in your code, with the exception that instead of a yield at the end, you add elements into the queue

def binarize(num_of_chunks, q):     ''' Generator function that yields chunks of binary data. In reality it wouldn't be the same data'''      ordered_col_list = ('col1', 'col2')     columns = dict.fromkeys(ordered_col_list)     for i in range(num_of_chunks):         columns['col1'] = array.array('i', range(10000)).tostring()         columns['col2'] = array.array('f', [float(num) for num in range(10000)]).tostring()         data = b''.join((columns[col_name] for col_name in ordered_col_list))         q.put(data) 

send_data should just be the while loop from the bottom of your code, with the connection open/close functionality

def send_data(ip, addr, q):      s = socket.create_connection((ip, addr))      while True:          try:              binary_chunk = q.get(False)          except:              sleep(0.1)              continue          else:                  socket.sendall(binary_chunk)              socket.recv(1000) # Get confirmation     # maybe remember to close the socket before killing the process 

Now you have two (three actually if you count the parent) processes that are processing data independently. You can force the two processes to synchronise their operations by setting the max_size of the queue to a single element. The operation of these two separate processes is also easy to monitor from the process manager on your computer top (Linux), Activity Monitor (OsX), don't remember what it's called under Windows.


Finally, Python 3 comes with the option of using co-routines which are neither processes nor threads, but something else entirely. Co-routines are pretty cool from a CS point of view, but a bit of a head scratcher at first. There is plenty of resources to learn from though, like this post on Medium and this talk by David Beazley.


Even more generally, you might want to look into the producer/consumer pattern, if you are not already familiar with it.

Read More

Monday, December 25, 2017

3-way handshake and get request using scapy in python

Leave a Comment

I am using scapy for 3-way handshake and sending get request and receiving response. But I am getting a TCP packet in response with FIN flag set. I am expecting HTTP packet with requested page. Where am I going wrong ?

import sys import socket  from scapy.all import *   # 3 way handshake ip=IP(dst="webs.com") SYN=TCP(sport=80, flags="S", seq=100, dport=80) SYNACK=sr1(ip/SYN)  my_ack = SYNACK.seq + 1 ACK=TCP(sport=80, flags="A", seq=101, ack=my_ack, dport=80) send(ip/ACK)  # request  PUSH = TCP(sport=80, dport=80, flags='PA', seq=102, ack=my_ack) payload = "GET / HTTP/1.1\r\nHost: webs.com\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.8\r\n\r\n" reply= sr1(ip/PUSH/payload, timeout=10) 

Wireshark result

wireshark result

2 Answers

Answers 1

Your machine is sending a RST packet. The RST packet is being sent by the kernel.
http://www.packetlevel.ch/html/scapy/scapy3way.html.

Try dropping the RST packet through iptables.
iptables -A OUTPUT -p tcp --tcp-flags RST RST -s 192.168.43.119 -j DROP

Answers 2

Looks like you're using a wrong sequence number when you send the request:

PUSH = TCP(sport=80, dport=80, flags='PA', seq=11, ack=my_ack) 

seq should be 101, not 11, since you used 100 for SYN. Changing it seems to fix the problem.

Also if you do not change the source port in your tests and you do not shut down the TCP connection properly or do not wait for 120 seconds between your tests the server might consider the new packets to belong to a previous connection and send something that you do not expect in response (depends on the state of the server connection).

Read More

Sunday, October 29, 2017

Communication in Netty Nio java

Leave a Comment

I want to create a communication system with two clients and a server in Netty nio. More specifically, firstly, I want when two clients are connected with the server to send a message from the server and after that to be able to exchnage data between the two clients. I am using the code provided from this example. My modifications in the code can be found here: link

It seems that the channelRead in the serverHandler works when the first client is connceted so it always return 1 but when a second client is connected does not change to 2. How can I check properly from the server when both clients are connected to the server? How can I read this value dynamically from my main function of the Client? Then which is the best way to let both clients communicate?

EDIT1: Apparently it seems that the client service is running and close directly so every time that I am running a new NettyClient is connected but the connection is closed after that. So the counter is always chnages from zero to one. As I was advised in the below comments I tested it using telnet in the same port and the counter seems to increasing normally, however, with the NettyClient service no.

EDIT2: It seems that the issue I got was from future.addListener(ChannelFutureListener.CLOSE); which was in channelRead in the ProcessingHandler class. When I commented it that out it seems that the code works. However, am not sure what are the consequences of commented that out. Moreover, I want from my main function of the client to check when the return message is specific two. How, could I create a method that waits for a specific message from server and meanwhile it blocks the main functionality.

 static EventLoopGroup workerGroup = new NioEventLoopGroup();  static Promise<Object> promise = workerGroup.next().newPromise();   public static void callClient() throws Exception {     String host = "localhost";     int port = 8080;     try {         Bootstrap b = new Bootstrap();         b.group(workerGroup);         b.channel(NioSocketChannel.class);         b.option(ChannelOption.SO_KEEPALIVE, true);         b.handler(new ChannelInitializer<SocketChannel>() {             @Override             public void initChannel(SocketChannel ch) throws Exception {                 ch.pipeline().addLast(new RequestDataEncoder(), new ResponseDataDecoder(), new ClientHandler(promise));             }         });         ChannelFuture f = b.connect(host, port).sync();     } finally {         //workerGroup.shutdownGracefully();     } } 

I want inside the main function to call the method and return the result and when it is 2 to continue with the main functionality. However, I cannot call callClient inside the while since it will run multiple times the same client.

   callBack();     while (true) {         Object msg = promise.get();         System.out.println("Case1: the connected clients is not two");         int ret = Integer.parseInt(msg.toString());         if (ret == 2){             break;         }     }    System.out.println("Case2: the connected clients is two");    // proceed with the main functionality 

How can I update the promise variable for the first client. When I run two clients, for the first client I always received the message :

Case1: the connected clients is not two

seems that the promise is not updated normally, while for the second client I always received the:

Case2: the connected clients is two

2 Answers

Answers 1

If my memory is correct, ChannelHandlerContext is one per channel and it can have multiple ChannelHandlers in it's pipeline. Your channels variable is an instance variable of your handler class. And you create a new ProcessingHandler instance for each connection. Thus each will have one and only one connection in channels variable once initialized - the one it was created for.

See new ProcessingHandler() in initChannel function in the server code (NettyServer.java).

You can either make channels variable static so that it is shared between ProcessingHandler instances. Or you can create a single ProcessingHandler instance elsewhere (e.g. as a local variable in the run() function) and then pass that instance to addLast call instead of new ProcessingHandler().

Answers 2

Why the size of ChannelGroup channels is always one. Even if I connect more clients?

Because child ChannelInitializer is called for every new Channel (client). There you are creating new instance of ProcessingHandler, so every channel see its own instance of ChannelGroup.

Solution 1 - Channel Attribute

Use Attribute and associate it with Channel.

Create attribute somewhere (let's say inside Constants class):

public static final AttributeKey<ChannelGroup> CH_GRP_ATTR =         AttributeKey.valueOf(SomeClass.class.getName()); 

Now, create ChannelGroup which will be used by all instances of ProcessingHandler:

final ChannelGroup channels = new DefaultChannelGroup(GlobalEventExecutor.INSTANCE); 

Update your child ChannelInitializer in NettyServer :

@Override public void initChannel(SocketChannel ch) throws Exception {     ch.pipeline().addLast(         new RequestDecoder(),          new ResponseDataEncoder(),          new ProcessingHandler());      ch.attr(Constants.CH_GRP_ATTR).set(channels); } 

Now you can access instance of ChannelGroup inside your handlers like this:

@Override public void channelActive(ChannelHandlerContext ctx) throws Exception {     final ChannelGroup channels = ctx.channel().attr(Constants.CH_GRP_ATTR).get();     channels.add(ctx.channel()); 

This will work, because every time new client connects, ChannelInitializer will be called with same reference to ChannelGroup.

Solution 2 - static field

If you declare ChannelGroup as static, all class instances will see same ChannelGroup instance:

private static final ChannelGroup channels =      new DefaultChannelGroup(GlobalEventExecutor.INSTANCE); 

Solution 3 - propagate shared instance

Introduce parameter into constructor of ProcessingHandler:

private final ChannelGroup channels; public ProcessingHandler(ChannelGroup chg) {     this.channels = chg; } 

Now, inside your NettyServer class create instance of ChannelGroup and propagate it to ProcessingHandler constructor:

final ChannelGroup channels = new        DefaultChannelGroup(GlobalEventExecutor.INSTANCE);  @Override public void initChannel(SocketChannel ch) throws Exception {     ch.pipeline().addLast(         new RequestDecoder(),          new ResponseDataEncoder(),          new ProcessingHandler(channels)); // <- here } 

Personally, I would choose first solution, because

  • It clearly associate ChannelGroup with Channel context
  • You can access same ChannelGroup in other handlers
  • You can have multiple instances of server (running on different port, within same JVM)
Read More

Sunday, October 22, 2017

Socket.io not sharing sessions with express

Leave a Comment

I can't figure why this is happening exactly since it was working perfectly before.

I'm using the following libraries:

"express-socket.io-session": "^1.3.2" "socket.io": "^2.0.3" "express": "^4.15.4" "express-session": "^1.15.5" 

1 - I login the user via a http request and send the cookie back to the frontend. All operations on http work perfectly with the cookie in frontend backend exchanges.

2 - After the user is logged in I tell the frontend "ok, user is logged in, now connect to the sockets":

io ( this.url ); 

Here is the relevant code:

var io_session = require("express-socket.io-session"); var e_session = require("express-session");  var sessionFileStore = require('session-file-store')(e_session);  var ee_session = e_session({     store: new sessionFileStore({ path: './user_sessions' }),     secret: "something-random",     resave: true,     saveUninitialized: true });  var enableCORS = function(req, res, next) {     res.header('Access-Control-Allow-Origin', req.headers.origin);     res.header('Access-Control-Allow-Methods', 'GET,PUT,POST,DELETE,OPTIONS');     res.header('Access-Control-Allow-Headers', 'Origin, Accept, Content-Type, Authorization, Content-Length, X-Requested-With, *');     res.header('Access-Control-Allow-Credentials',true);          // intercept OPTIONS method     if ('OPTIONS' == req.method) {         res.sendStatus(200);     } else {         next();     }; };  app.use(function(req, res, next) {         ///    next(); });  //app.use(cookieParser()); app.use(enableCORS); app.use(bodyParser.urlencoded({ extended: true })); app.use(bodyParser.json()); app.use(ee_session);  preparedApp = require("http").Server(app);  var io = require("socket.io")(preparedApp);  io.use(io_session(e_session,{     autoSave: true }));  preparedApp.listen(8080, function(){});  io.on('connection', function(socket){      var socket_session = socket.handshake.session,         socket_session_id = socket.handshake.sessionID;      console.log("SOCKET_SESSION:",socket_session);     console.log("SOCKET_SESSION_ID:",socket_session_id);      (...) 

3 - socket_session is 'empty' but everything in the regular http cookies works. The session is maintained there.

One thing I noticed is that socket_session_id points to a session file that does not exist inside the folder user_sessions. The only ones that exist are created in the http login. So, basically:

=> User logins: efihaeif939311kf3f3 session id file is created.

=> Socket connects: fiaejgieofaekofek is the session id in the same login flow but file does not exist in user_sessions (note that the session id is not the same)

Any idea on why this is happening? I have absolutely no idea.

Thanks

1 Answers

Answers 1

try sharing the session by stringifying the session json and retrieving and parsing back to session object in the client side.

This might work.

Read More

Monday, October 2, 2017

Why does the Linux kernel have `struct sock` and `struct socket`?

Leave a Comment

This questions was asked before on the Internet, but I couldn't find a good answer.

The Linux kernel networking stack features two structures:

The two structures are essentially linked, but seem to have slightly different lifetimes. One can find an sk via sock->sk, or find a sock via sk->sk_socket.

Why are there two structures to store information about sockets? Assuming I need to add a new field, when would I add it to struct socket and when to struct sock?

UPDATE: Please note that I refer to struct socket in include/linux/net.h inside the Linux source code, which is meant for kernel code only, and not /usr/include/sys/socket.h which is meant for userland.

2 Answers

Answers 1

struct socket seems to be a higher level interface that is used for system calls (that is why it also has pointer to struct file which represents file descriptor here).

struct sock is a in-kernel implemenation for AF_INET sockets (there is also struct unix_sock for AF_UNIX sockets which is derivative of this) which can be used both by kernel and by userspace (via struct sock).

Both were added to Linux 1.0 back in 1993, I doubt you'll find a doc specifying initial design decision.

Answers 2

“The two structures are essentially linked” - not sure what you meant.

I guess you could find answer if look at source files for these structures:

socket  -> linux-src/include/linux/net.h sock    -> linux-src/include/net/sock.h 

socket

 * NET      An implementation of the SOCKET network access protocol.  *      This is the master header file for the Linux NET layer,  *      or, in plain English: the networking handling part of the  *      kernel. 

sock

 * INET     An implementation of the TCP/IP protocol suite for the LINUX  *      operating system.  INET is implemented using the  BSD Socket  *      interface as the means of communication with the user level. 

These structures are different and has different representation of socket abstraction.

Here answer about different sockets.

Unix vs BSD vs TCP vs Internet sockets?

Where to define additional field depends on your intention. Please describe your task.

Please look at sources.

linux-src/include/linux/net.h

/*  * NET      An implementation of the SOCKET network access protocol.  *      This is the master header file for the Linux NET layer,  *      or, in plain English: the networking handling part of the  *      kernel.  *  * Version: @(#)net.h   1.0.3   05/25/93  *  * Authors: Orest Zborowski, <obz@Kodak.COM>  *      Ross Biro  *      Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>  *  *      This program is free software; you can redistribute it and/or  *      modify it under the terms of the GNU General Public License  *      as published by the Free Software Foundation; either version  *      2 of the License, or (at your option) any later version.  */ ..... ..... ..... /**  *  struct socket - general BSD socket  *  @state: socket state (%SS_CONNECTED, etc)  *  @type: socket type (%SOCK_STREAM, etc)  *  @flags: socket flags (%SOCK_NOSPACE, etc)  *  @ops: protocol specific socket operations  *  @file: File back pointer for gc  *  @sk: internal networking protocol agnostic socket representation  *  @wq: wait queue for several uses  */ struct socket {     socket_state        state;      kmemcheck_bitfield_begin(type);     short           type;     kmemcheck_bitfield_end(type);      unsigned long       flags;      struct socket_wq __rcu  *wq;      struct file     *file;     struct sock     *sk;     const struct proto_ops  *ops; }; 

linux-src/include/net/sock.h

/*  * INET     An implementation of the TCP/IP protocol suite for the LINUX  *      operating system.  INET is implemented using the  BSD Socket  *      interface as the means of communication with the user level.  *  *      Definitions for the AF_INET socket handler.  *  * Version: @(#)sock.h  1.0.4   05/13/93  *  * Authors: Ross Biro  *      Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>  *      Corey Minyard <wf-rch!minyard@relay.EU.net>  *      Florian La Roche <flla@stud.uni-sb.de>  *  * Fixes:  *      Alan Cox    :   Volatiles in skbuff pointers. See  *                  skbuff comments. May be overdone,  *                  better to prove they can be removed  *                  than the reverse.  *      Alan Cox    :   Added a zapped field for tcp to note  *                  a socket is reset and must stay shut up  *      Alan Cox    :   New fields for options  *  Pauline Middelink   :   identd support  *      Alan Cox    :   Eliminate low level recv/recvfrom  *      David S. Miller :   New socket lookup architecture.  *              Steve Whitehouse:       Default routines for sock_ops  *              Arnaldo C. Melo :   removed net_pinfo, tp_pinfo and made  *                          protinfo be just a void pointer, as the  *                          protocol specific parts were moved to  *                          respective headers and ipv4/v6, etc now  *                          use private slabcaches for its socks  *              Pedro Hortas    :   New flags field for socket options  *  *  *      This program is free software; you can redistribute it and/or  *      modify it under the terms of the GNU General Public License  *      as published by the Free Software Foundation; either version  *      2 of the License, or (at your option) any later version.  */ .... .... .... /**   * struct sock - network layer representation of sockets   * @__sk_common: shared layout with inet_timewait_sock   * @sk_shutdown: mask of %SEND_SHUTDOWN and/or %RCV_SHUTDOWN   * @sk_userlocks: %SO_SNDBUF and %SO_RCVBUF settings   * @sk_lock:   synchronizer   * @sk_kern_sock: True if sock is using kernel lock classes   * @sk_rcvbuf: size of receive buffer in bytes   * @sk_wq: sock wait queue and async head   * @sk_rx_dst: receive input route used by early demux   * @sk_dst_cache: destination cache   * @sk_dst_pending_confirm: need to confirm neighbour   * @sk_policy: flow policy   * @sk_receive_queue: incoming packets   * @sk_wmem_alloc: transmit queue bytes committed   * @sk_tsq_flags: TCP Small Queues flags   * @sk_write_queue: Packet sending queue   * @sk_omem_alloc: "o" is "option" or "other"   * @sk_wmem_queued: persistent queue size   * @sk_forward_alloc: space allocated forward   * @sk_napi_id: id of the last napi context to receive data for sk   * @sk_ll_usec: usecs to busypoll when there is no data   * @sk_allocation: allocation mode   * @sk_pacing_rate: Pacing rate (if supported by transport/packet scheduler)   * @sk_pacing_status: Pacing status (requested, handled by sch_fq)   * @sk_max_pacing_rate: Maximum pacing rate (%SO_MAX_PACING_RATE)   * @sk_sndbuf: size of send buffer in bytes   * @__sk_flags_offset: empty field used to determine location of bitfield   * @sk_padding: unused element for alignment   * @sk_no_check_tx: %SO_NO_CHECK setting, set checksum in TX packets   * @sk_no_check_rx: allow zero checksum in RX packets   * @sk_route_caps: route capabilities (e.g. %NETIF_F_TSO)   * @sk_route_nocaps: forbidden route capabilities (e.g NETIF_F_GSO_MASK)   * @sk_gso_type: GSO type (e.g. %SKB_GSO_TCPV4)   * @sk_gso_max_size: Maximum GSO segment size to build   * @sk_gso_max_segs: Maximum number of GSO segments   * @sk_lingertime: %SO_LINGER l_linger setting   * @sk_backlog: always used with the per-socket spinlock held   * @sk_callback_lock: used with the callbacks in the end of this struct   * @sk_error_queue: rarely used   * @sk_prot_creator: sk_prot of original sock creator (see ipv6_setsockopt,   *           IPV6_ADDRFORM for instance)   * @sk_err: last error   * @sk_err_soft: errors that don't cause failure but are the cause of a   *           persistent failure not just 'timed out'   * @sk_drops: raw/udp drops counter   * @sk_ack_backlog: current listen backlog   * @sk_max_ack_backlog: listen backlog set in listen()   * @sk_uid: user id of owner   * @sk_priority: %SO_PRIORITY setting   * @sk_type: socket type (%SOCK_STREAM, etc)   * @sk_protocol: which protocol this socket belongs in this network family   * @sk_peer_pid: &struct pid for this socket's peer   * @sk_peer_cred: %SO_PEERCRED setting   * @sk_rcvlowat: %SO_RCVLOWAT setting   * @sk_rcvtimeo: %SO_RCVTIMEO setting   * @sk_sndtimeo: %SO_SNDTIMEO setting   * @sk_txhash: computed flow hash for use on transmit   * @sk_filter: socket filtering instructions   * @sk_timer: sock cleanup timer   * @sk_stamp: time stamp of last packet received   * @sk_tsflags: SO_TIMESTAMPING socket options   * @sk_tskey: counter to disambiguate concurrent tstamp requests   * @sk_zckey: counter to order MSG_ZEROCOPY notifications   * @sk_socket: Identd and reporting IO signals   * @sk_user_data: RPC layer private data   * @sk_frag: cached page frag   * @sk_peek_off: current peek_offset value   * @sk_send_head: front of stuff to transmit   * @sk_security: used by security modules   * @sk_mark: generic packet mark   * @sk_cgrp_data: cgroup data for this cgroup   * @sk_memcg: this socket's memory cgroup association   * @sk_write_pending: a write to stream socket waits to start   * @sk_state_change: callback to indicate change in the state of the sock   * @sk_data_ready: callback to indicate there is data to be processed   * @sk_write_space: callback to indicate there is bf sending space available   * @sk_error_report: callback to indicate errors (e.g. %MSG_ERRQUEUE)   * @sk_backlog_rcv: callback to process the backlog   * @sk_destruct: called at sock freeing time, i.e. when all refcnt == 0   * @sk_reuseport_cb: reuseport group container   * @sk_rcu: used during RCU grace period   */ struct sock {     /*      * Now struct inet_timewait_sock also uses sock_common, so please just      * don't add nothing before this first member (__sk_common) --acme      */     struct sock_common  __sk_common; #define sk_node         __sk_common.skc_node #define sk_nulls_node       __sk_common.skc_nulls_node #define sk_refcnt       __sk_common.skc_refcnt #define sk_tx_queue_mapping __sk_common.skc_tx_queue_mapping  #define sk_dontcopy_begin   __sk_common.skc_dontcopy_begin #define sk_dontcopy_end     __sk_common.skc_dontcopy_end #define sk_hash         __sk_common.skc_hash #define sk_portpair     __sk_common.skc_portpair #define sk_num          __sk_common.skc_num #define sk_dport        __sk_common.skc_dport #define sk_addrpair     __sk_common.skc_addrpair #define sk_daddr        __sk_common.skc_daddr #define sk_rcv_saddr        __sk_common.skc_rcv_saddr #define sk_family       __sk_common.skc_family #define sk_state        __sk_common.skc_state #define sk_reuse        __sk_common.skc_reuse #define sk_reuseport        __sk_common.skc_reuseport #define sk_ipv6only     __sk_common.skc_ipv6only #define sk_net_refcnt       __sk_common.skc_net_refcnt #define sk_bound_dev_if     __sk_common.skc_bound_dev_if #define sk_bind_node        __sk_common.skc_bind_node #define sk_prot         __sk_common.skc_prot #define sk_net          __sk_common.skc_net #define sk_v6_daddr     __sk_common.skc_v6_daddr #define sk_v6_rcv_saddr __sk_common.skc_v6_rcv_saddr #define sk_cookie       __sk_common.skc_cookie #define sk_incoming_cpu     __sk_common.skc_incoming_cpu #define sk_flags        __sk_common.skc_flags #define sk_rxhash       __sk_common.skc_rxhash      socket_lock_t       sk_lock;     atomic_t        sk_drops;     int         sk_rcvlowat;     struct sk_buff_head sk_error_queue;     struct sk_buff_head sk_receive_queue;     /*      * The backlog queue is special, it is always used with      * the per-socket spinlock held and requires low latency      * access. Therefore we special case it's implementation.      * Note : rmem_alloc is in this structure to fill a hole      * on 64bit arches, not because its logically part of      * backlog.      */     struct {         atomic_t    rmem_alloc;         int     len;         struct sk_buff  *head;         struct sk_buff  *tail;     } sk_backlog; #define sk_rmem_alloc sk_backlog.rmem_alloc      int         sk_forward_alloc; #ifdef CONFIG_NET_RX_BUSY_POLL     unsigned int        sk_ll_usec;     /* ===== mostly read cache line ===== */     unsigned int        sk_napi_id; #endif     int         sk_rcvbuf;      struct sk_filter __rcu  *sk_filter;     union {         struct socket_wq __rcu  *sk_wq;         struct socket_wq    *sk_wq_raw;     }; #ifdef CONFIG_XFRM     struct xfrm_policy __rcu *sk_policy[2]; #endif     struct dst_entry    *sk_rx_dst;     struct dst_entry __rcu  *sk_dst_cache;     atomic_t        sk_omem_alloc;     int         sk_sndbuf;      /* ===== cache line for TX ===== */     int         sk_wmem_queued;     refcount_t      sk_wmem_alloc;     unsigned long       sk_tsq_flags;     struct sk_buff      *sk_send_head;     struct sk_buff_head sk_write_queue;     __s32           sk_peek_off;     int         sk_write_pending;     __u32           sk_dst_pending_confirm;     u32         sk_pacing_status; /* see enum sk_pacing */     long            sk_sndtimeo;     struct timer_list   sk_timer;     __u32           sk_priority;     __u32           sk_mark;     u32         sk_pacing_rate; /* bytes per second */     u32         sk_max_pacing_rate;     struct page_frag    sk_frag;     netdev_features_t   sk_route_caps;     netdev_features_t   sk_route_nocaps;     int         sk_gso_type;     unsigned int        sk_gso_max_size;     gfp_t           sk_allocation;     __u32           sk_txhash;      /*      * Because of non atomicity rules, all      * changes are protected by socket lock.      */     unsigned int        __sk_flags_offset[0]; #ifdef __BIG_ENDIAN_BITFIELD #define SK_FL_PROTO_SHIFT  16 #define SK_FL_PROTO_MASK   0x00ff0000  #define SK_FL_TYPE_SHIFT   0 #define SK_FL_TYPE_MASK    0x0000ffff #else #define SK_FL_PROTO_SHIFT  8 #define SK_FL_PROTO_MASK   0x0000ff00  #define SK_FL_TYPE_SHIFT   16 #define SK_FL_TYPE_MASK    0xffff0000 #endif      kmemcheck_bitfield_begin(flags);     unsigned int        sk_padding : 1,                 sk_kern_sock : 1,                 sk_no_check_tx : 1,                 sk_no_check_rx : 1,                 sk_userlocks : 4,                 sk_protocol  : 8,                 sk_type      : 16; #define SK_PROTOCOL_MAX U8_MAX     kmemcheck_bitfield_end(flags);      u16         sk_gso_max_segs;     unsigned long           sk_lingertime;     struct proto        *sk_prot_creator;     rwlock_t        sk_callback_lock;     int         sk_err,                 sk_err_soft;     u32         sk_ack_backlog;     u32         sk_max_ack_backlog;     kuid_t          sk_uid;     struct pid      *sk_peer_pid;     const struct cred   *sk_peer_cred;     long            sk_rcvtimeo;     ktime_t         sk_stamp;     u16         sk_tsflags;     u8          sk_shutdown;     u32         sk_tskey;     atomic_t        sk_zckey;     struct socket       *sk_socket;     void            *sk_user_data; #ifdef CONFIG_SECURITY     void            *sk_security; #endif     struct sock_cgroup_data sk_cgrp_data;     struct mem_cgroup   *sk_memcg;     void            (*sk_state_change)(struct sock *sk);     void            (*sk_data_ready)(struct sock *sk);     void            (*sk_write_space)(struct sock *sk);     void            (*sk_error_report)(struct sock *sk);     int         (*sk_backlog_rcv)(struct sock *sk,                           struct sk_buff *skb);     void                    (*sk_destruct)(struct sock *sk);     struct sock_reuseport __rcu *sk_reuseport_cb;     struct rcu_head     sk_rcu; }; 
Read More

Thursday, September 14, 2017

Can I find out the status of the port using the Lua “socket” library?

Leave a Comment

Help me track the status of a specific port: "LISTENING", "CLOSE_WAIT", "ESTABLISHED". I have an analog solution with the netstat command:

local command = 'netstat -anp tcp | find ":1926 " ' local h = io.popen(command,"rb") local result = h:read("*a") h:close() print(result) if result:find("ESTABLISHED") then    print("Ok") end 

But I need to do the same with the Lua socket library. Is it possible?

2 Answers

Answers 1

Like @Peter said, netstat uses the proc file system to gather network information, particularly port bindings. LuaSockets has it's own library to retrieve connection information. For example,

Listening you can use master:listen(backlog) which specifies the socket is willing to receive connections, transforming the object into a server object. Server objects support the accept, getsockname, setoption, settimeout, and close methods. The parameter backlog specifies the number of client connections that can be queued waiting for service. If the queue is full and another client attempts connection, the connection is refused. In case of success, the method returns 1. In case of error, the method returns nil followed by an error message.

The following methods will return a string with the local IP address and a number with the port. In case of error, the method returns nil.

master:getsockname() client:getsockname() server:getsockname() 

There also exists this method: client:getpeername() That will return a string with the IP address of the peer, followed by the port number that peer is using for the connection. In case of error, the method returns nil.

For "CLOSE_WAIT", "ESTABLISHED", or other connection information you want to retrieve, please read the Official Documentation. It has everything you need with concise explanations of methods.

Answers 2

You can't query the status of a socket owned by another process using the sockets API, which is what LuaSocket uses under the covers.

In order to access information about another process, you need to query the OS instead. Assuming you are on Linux, this usually means looking at the proc filesystem.

I'm not hugely familiar with Lua, but a quick Google gives me this project: https://github.com/Wiladams/lj2procfs. I think this is probably what you need, assuming they have written a decoder for the relevant /proc/net files you need.

As for which file? If it's just the status, I think you want the tcp file as covered in http://www.onlamp.com/pub/a/linux/2000/11/16/LinuxAdmin.html

Read More

Friday, September 8, 2017

PEAR Mail unable to connect to Gmail SMTP, failed to connect to socket

Leave a Comment

Facts

I am using PEAR Mail, I want to use gmail SMTP to send a mail. I have Apache/2.4.27 (Win64) PHP/7.2.0beta3, PEAR 1.10.15, Mail 1.4.1, Net_SMTP 1.8.0, Net_Socket 1.2.2.

I went to php.ini and added extension = php_openssl.dll. The error.log gives no ssl-related errors.

Here is the code

require_once "Mail.php";  $from = '<slevin@gmail.com>'; $to = '<slevinkelevra@gmal.com>'; $subject = 'Hi!'; $body = "Hi,\n\nHow are you?";  $headers = array(     'From' => $from,     'To' => $to,     'Subject' => $subject );  $smtp = Mail::factory('smtp', array(         'host' => 'ssl://smtp.gmail.com',         'port' => '465',         'auth' => true,         'username' => 'slevinmail@gmail.com',         'password' => 'mypassword'     ));  $mail = $smtp->send($to, $headers, $body);  if (PEAR::isError($mail)) {     echo('<p>' . $mail->getMessage() . '</p>'); } else {     echo('<p>Message successfully sent!</p>'); } 

The problem

I get this error

Failed to connect to ssl://smtp.gmail.com:465 [SMTP: Failed to connect socket: fsockopen(): unable to connect to ssl://smtp.gmail.com:465 (Unknown error) (code: -1, response: )]

and I have no clue what to do, I Googled but I got more confused.

Please advice on how to fix this. Thank you

Update

Following symcbean's instructions I got the following results :

bool(true)   array(5) {  [0]=> string(31) "alt3.gmail-smtp-in.l.google.com"  [1]=> string(26) "gmail-smtp-in.l.google.com"  [2]=> string(31) "alt4.gmail-smtp-in.l.google.com"  [3]=> string(31) "alt1.gmail-smtp-in.l.google.com"  [4]=> string(31) "alt2.gmail-smtp-in.l.google.com" }  IPV4 address = 64.233.188.27  If you've got this far without errors then problem is with your SSL config  Check you've got your cacerts deployed in one of the following locations default_cert_file = C:\Program Files\Common Files\SSL/cert.pem default_cert_file_env = SSL_CERT_FILE default_cert_dir = C:\Program Files\Common Files\SSL/certs default_cert_dir_env = SSL_CERT_DIR default_private_dir = C:\Program Files\Common Files\SSL/private default_default_cert_area = C:\Program Files\Common Files\SSL ini_cafile =  ini_capath =   If all good so far, then this bit should work.... fsockopen  Warning: fsockopen(): SSL operation failed with code 1. OpenSSL Error messages: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed in C:\Apache24\htdocs\phptest2.php on line 28  Warning: fsockopen(): Failed to enable crypto in C:\Apache24\htdocs\phptest2.php on line 28  Warning: fsockopen(): unable to connect to ssl://smtp.gmail.com:465 (Unknown error) in C:\Apache24\htdocs\phptest2.php on line 28 bool(false) int(0) string(0) ""  

Line 28 is this line var_dump(fsockopen("ssl://smtp.gmail.com", 465, $errno, $errstr, 3.0));

Thanks again

Update #2

I googled just "fsockopen(): SSL operation failed with code 1." of the first warning.

End up here . I changed the mail port of the AVG, like the answer. symcbean's code run with no errors, but my code replied with mail error : authentication failure [SMTP: Invalid response code received from server (code: 534, response: 5.7.14 Please log in via your web browser and 5.7.14 then try again. 5.7.14 Learn more at 5.7.14 https://support.google.com/mail/answer/78754 c1sm1243434wre.84 - gsmtp)]

So I googled code: 534, response: 5.7.14 and end-up here, followed the instructions of the first answer by emgh3i, enabled less secured connections and allowed access to my google account

And its working perfectly now.

7 Answers

Answers 1

Few debugging steps :

1. check phpinfo

I recommend checking phpinfo() to check whether all modules are enabled. Check for mail, fsocketopen.

2. Enable debug flag

Enable debug flag to check exactly what's the problem. Like below.

$smtp = Mail::factory('smtp', array(         'host' => 'ssl://smtp.gmail.com',         'port' => '465',         'auth' => true,         'debug' => true,         'pipelining' => true,         'username' => 'xxx@gmail.com',         'password' => 'xxx'     )); 

After running above code on my machine I got follow response. Issue can be different from yours. But debug helped me. As I have 2FA enabled, it gave me error. And I got a mail also, that my login has been blocked.

DEBUG: Recv: 220 smtp.gmail.com ESMTP s65sm4891344pfi.36 - gsmtp DEBUG: Send: EHLO localhost DEBUG: Recv: 250-smtp.gmail.com at your service, [110.227.210.84] DEBUG: Recv: 250-SIZE 35882577 DEBUG: Recv: 250-8BITMIME DEBUG: Recv: 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH DEBUG: Recv: 250-ENHANCEDSTATUSCODES DEBUG: Recv: 250-PIPELINING DEBUG: Recv: 250-CHUNKING DEBUG: Recv: 250 SMTPUTF8 DEBUG: Send: AUTH LOGIN DEBUG: Recv: 334 VsadfSFcm5hbWU6 DEBUG: Send: cGF0ZWwuZ29wYhkafdaASFnbWFpbC5jb20= DEBUG: Recv: 334 UGFzc3dvcmQ6 DEBUG: Send: OWwzMy5zaHlAbTE4 DEBUG: Recv: 534-5.7.14 Please log in via your web browser and DEBUG: Recv: 534-5.7.14 then try again. DEBUG: Recv: 534-5.7.14 Learn more at DEBUG: Recv: 534 5.7.14 https://support.google.com/mail/answer/78754 s65sm4891344pfi.36 - gsmtp DEBUG: Send: RSET DEBUG: Send: QUIT DEBUG: Recv: 250 2.1.5 Flushed s65sm4891344pfi.36 - gsmtp DEBUG: Recv: 221 2.0.0 closing connection s65sm4891344pfi.36 - gsmtp authentication failure [SMTP: Invalid response code received from server (code: 534, response: 5.7.14 Please log in via your web browser and 5.7.14 then try again. 5.7.14 Learn more at 5.7.14 https://support.google.com/mail/answer/78754 s65sm4891344pfi.36 - gsmtp)] 

Update:

Your issue looks like PHP is not even able to connect to gmail server.

Answers 2

Your host configuration shouldn't contain the protocol. The reason it's failing is because it's probably trying to perform a DNS Lookup on ssl://smtp.gmail.com and failing.

Change

'host' => 'ssl://smtp.gmail.com', 

to

'host' => 'smtp.gmail.com', 

Answers 3

Your code is correct

I tried to test my gmail account. Mail sending was successful.

Check your socket connection

<?php  error_reporting(E_ALL);  var_dump(fsockopen("ssl://smtp.gmail.com", 465, $errno, $errstr)); var_dump($errno); var_dump($errstr); 

resource(4) of type (stream)

int(0)

string(0) ""

Answers 4

Been Kyung-yoong is the only person to have made a meaningful contribution to solving the problem so far (+1 Been!). I can confirm his result. And I would recommend you try the same. You are currently trying to debug a rather complex stack of components:

Been is doing your job for you - as the person posting the question - should be creating a Minimal, Complete, and Verifiable example

This will hopefully also provide more meaningful diagnostic information.

The most likely reasons for this to be failing are:

  • the host you are running this on cannot route outgoing connections to the internet (but since you seem to be using a desktop PC, I would think you might have noticed this by now)
  • the code is running within a security sandbox (but MSWindows doesn't really have such things)
  • the host is unable to resolve the hostname (see first point about routing)
  • the host is able to connect but unable to verify the certificate

Hence you might consider this more elaborate implementation of a test script:

 <?php   error_reporting(E_ALL);   print "DNS\n";  var_dump(getmxrr('gmail.com',$result));  var_dump($result);  $use_ip=gethostbyname($result[0]);  print "IPV4 address = $use_ip\n";   print "\nIf you've got this far without errors then problem is with your SSL config\n";  $calocns=openssl_get_cert_locations();  if (count($calocns)) {      print "Check you've got your cacerts deployed in one of the following locations\n";      foreach ($calocns as $k=>$v) print "$k = $v\n";  } else {      print "You've not configured your openssl installation on this host\n";  }   print "\nIf all good so far, then this bit should work....\n";  print "fsockopen\n";  var_dump(fsockopen("ssl://smtp.gmail.com", 465, $errno, $errstr, 3.0));  var_dump($errno);  var_dump($errstr); 

Which should give you a response like this:

 DNS  bool(true)  array(5) {    [0]=>    string(31) "alt1.gmail-smtp-in.l.google.com"    [1]=>    string(31) "alt2.gmail-smtp-in.l.google.com"    [2]=>    string(31) "alt4.gmail-smtp-in.l.google.com"    [3]=>    string(26) "gmail-smtp-in.l.google.com"    [4]=>    string(31) "alt3.gmail-smtp-in.l.google.com"  }  IPV4 address = 74.125.131.26   If you've got this far without errors then problem is with your SSL config  Check you've got your cacerts deployed in one of the following locations  default_cert_file = /usr/lib/ssl/cert.pem  default_cert_file_env = SSL_CERT_FILE  default_cert_dir = /usr/lib/ssl/certs  default_cert_dir_env = SSL_CERT_DIR  default_private_dir = /usr/lib/ssl/private  default_default_cert_area = /usr/lib/ssl  ini_cafile =  ini_capath =   If all good so far, then this bit should work....  fsockopen  resource(4) of type (stream)  int(0)  string(0) "" 

Given that we can't replicate your error we can't give a definitive answer what the problem is - but my guess would be that you haven't configure openSSL.

Answers 5

Before I begin, let me preface this that there are many possibly solutions and outcomes between your server and the google server, so these may or may not work for different people.

1) SMTP is not very secure, so Google may be rejecting your request. I had this problem 6 months ago and the solution was enabling insecure apps under 'myaccount.google.com'

enter image description here

2) If that doesn't work for you then you may consider switching protocols.

from

'host' => 'ssl://smtp.gmail.com',

to

'host' => 'tls://smtp.gmail.com:587';

Answers 6

  • The "Use the Gmail SMTP Server" section of this guide says you need to enable "Less secure apps".

  • I notice that the from address you are using for the message is not the same as the Gmail account you are using to send. I would guess that is a problem. Try using $from = '<slevinmail@gmail.com>';, so that address is the same one you are using in the 'username' => 'slevinmail@gmail.com', field.

Answers 7

When something fail and we don't know the cause we have to do debugging. So here instead of putting an answer I am requesting you to execute some tests

  1. confirm system connectivity with internet: Open cmd terminal and type

    ping smtp.gmail.com 
  2. confirm firewall: Enter following in cmd terminal

    telnet smtp.gmail.com 465 
  3. confirm php setup: enter php -a at cmd terminal and on php prompt execute (copy / paste and then press enter) following code.

    $result = fsockopen('ssl://smtp.gmail.com', 465, $error_no, $error_message, 5); if ($result === false) {   echo "error no: $error_no error message: $error_message";   echo print_r($result, true); } else {   echo 'success'; } 
  4. confirm Pear Mail library and Gmail SMTP access: again on cmd and php prompt php -a execute your own code (as you posted in this thread)

And lets know where it breaks, and what is the error. Only after that we can help

Read More

Wednesday, May 17, 2017

Django 1.10 & Socket.IO with Python 3

Leave a Comment

I'm trying to find some "django-socketio" repo to use in my project. I using django 1.10 and python3. I really searched but I do not found working examples with python3.

My poor workaround

  • I started node project and put socket.io inside route
  • In my django view I send returning data to node route with my django session
  • I manage session coming from django inside my node and emit inside route to client.

This work but I can't believe this is a good solution.. Anyone have other ideas? Or working examples with python3 and socketio?

Thanks!

1 Answers

Answers 1

If you want to use Websockets and Django you should consider https://github.com/django/channels. The alternative in Python would be using python tornado http://www.tornadoweb.org/en/stable/ or aiohttp (Python3.4+) http://aiohttp.readthedocs.io/en/stable/. Many of the implementations of Django with asynchronousity through gevent are outdated, experimental or abandoned, I found this https://github.com/jrief/django-websocket-redis but it uses Redis so no reason to not going back to django-channels.

In my opinion, as Socket.io is a layer over Websockets you will not find any project that supports fully the Socket.io spec in Python as it is a native Node.js not officially ported to Python project, at least the latest one you probably are using, if you really need Socket.io features stick to Node.js and create a simple REST API in Django to load the backend data synchronously, this is the best shot you would likely have.

Read More

Monday, May 15, 2017

Alamofire request always fails with “The request timed out” if Socket.io is connected

Leave a Comment

if socket.io is connected Alamofire is not working as expected, always getting req time out error. Alamofire works if i disable Socket.io

this is the error i'm getting

Error Domain=NSURLErrorDomain Code=-1001 "The request timed out."  UserInfo={NSUnderlyingError=0x60800044c720  {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)"  UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, NSErrorFailingURLStringKey=https://enpoint.json, NSErrorFailingURLKey=https://enpoint.json,  _kCFStreamErrorDomainKey=4, _kCFStreamErrorCodeKey=-2102, NSLocalizedDescription=The request timed out.} 

and this

_tcp_connection_write_eof_block_invoke Write close callback received error: [89] Operation canceled 

kind of similar issue reported on git https://github.com/Alamofire/Alamofire/issues/1545 but no answers.

I don't know what i'm doing wrong! Any help would be appreciated

1 Answers

Answers 1

I was able to fix this issue by changing alamofire parameter encoding. By default Alamofire sets encoding to URLEncoding if you are not passing any encoding argument. Use URLEncoding for GET and DELETE request. Use JSONEncoding.default for all other requests (API should have support for this).

Still i'm not sure why it was working w/o socket and wasn't working with socket on.

Read More

Thursday, April 13, 2017

Not getting response in socket connection

Leave a Comment

I cannot get response in a socket connection and I couldnt understand what is wrong with the code. I could able to establish a socket connection using the ip address and port number, and it is entering into

    if (nsocket.isConnected()) {}  

When I tried with telnet I could get the response . But the input has some other parameters like:

POST /setMap HTTP/1.1 Host: 192.168.1.1 Content-Type: application/json; charset=utf-8 Content-Length: 1234

{ "cmd":"request_get_file_list","verification":"CVS" }

I dont know how to include the connection properties like content type, length in my code.

Here is the code:

public class WebService {  public static String devicelisting() {     Socket nsocket;      String response = null;      try {         nsocket = new Socket("192.168.1.1", 6666);         if (nsocket.isConnected()) {              JSONObject json = new JSONObject();             json.put("cmd", "request_get_file_list");             json.put("verification", "CVS");             Log.i("AsyncTask", "doInBackground: Creating socket");             // nsocket = new Socket();              OutputStreamWriter out = new OutputStreamWriter(nsocket.getOutputStream(), StandardCharsets.UTF_8);                 out.write(json.toString());             Log.i("Webservice", "json.toString"+json.toString());              InputStream in = new BufferedInputStream(nsocket.getInputStream());             BufferedReader r = new BufferedReader(new InputStreamReader(in));             StringBuilder stringbuilder = new StringBuilder();             String line;             while ((line = r.readLine()) != null) {                 stringbuilder.append(line);                 Log.i("line", "line.line"+line);             }               response = stringbuilder.toString();             Log.i("Response", response);         }         else{             Log.i("Response", "not connected");          }      } catch (ProtocolException e) {         e.printStackTrace();     } catch (MalformedURLException e) {         e.printStackTrace();     } catch (UnknownHostException e) {         e.printStackTrace();     } catch (UnsupportedEncodingException e) {         e.printStackTrace();     } catch (IOException e) {         e.printStackTrace();     } catch (JSONException e) {         e.printStackTrace();     }     return response; } 

Please help me to find the issue. I am badly stuck up .Please help me resolve the issue

3 Answers

Answers 1

For socket driven events it is difficult to implement many functions while there are some (open source) libraries to achieve such a task. Consider using Socket.io.

Properties headers = new Properties(); headers.setProperty("Content-Type","application/json"); // your headers SocketIO socketIO = SocketIO(url, headers); 

For more information have a look at SocketIO docs

Edit

In your given example you should use HttpURLConnection as you are getting a response from server, you do not need to implement sockets. Simply GET or POST to fetch or push your data using HttpURLConnection.

Answers 2

For socket connection in android, you can use this gist file that simply implement socket connection.

public SocketConnection(OnStatusChanged statusChangedListener, OnMessageReceived messageReceivedListener) {         mStatusListener = statusChangedListener;         mMessageListener = messageReceivedListener;          isRunning = true;         try {             InetAddress serverAddr = InetAddress.getByName(SERVER_IP);             mStatusListener.statusChanged(WAITING);             socket = new Socket(serverAddr, SERVER_PORT);             try {                 printWriter = new PrintWriter(new BufferedWriter(new OutputStreamWriter(socket.getOutputStream())), true);                 mStatusListener.statusChanged(CONNECTED);                 bufferedReader = new BufferedReader(new InputStreamReader(socket.getInputStream()));                 while (isRunning) {                     retrieveMessage = bufferedReader.readLine();                     if (retrieveMessage != null && mMessageListener != null) {                         mMessageListener.messageReceived(retrieveMessage);                     }                     else {                         mStatusListener.statusChanged(DISCONNECTED);                     }                     retrieveMessage = null;                 }             } catch (Exception e) {                 mStatusListener.statusChanged(ERROR);             } finally {                 socket.close();             }         } catch (Exception e) {             mStatusListener.statusChanged(ERROR);         }     } 

Answers 3

The code is probably stuck in readLine() because the server still waits for the request's completion.

You could change it to :

// query out.write("POST /setMap HTTP/1.1\r\n"); // headers out.write("Host: 192.168.1.1\r\n"); out.write("Content-Type: application/json; charset=utf-8\r\n"); out.write("Content-Length: " + json.toString().getBytes(StandardCharsets.UTF_8).length + "\r\n"); // end of the headers out.write("\r\n"); // body out.write(json.toString()); // actually send the request out.flush();  Log.i("Webservice", "json.toString"+json.toString()); 

I think that you are using the wrong tool to make HTTP requests. Sockets are low level network channels, you have to do a lot of things yourself.

You should consider using an HttpURLConnection instead. If possible I strongly suggest to take a even higher level approach, and use something like retrofit2 for example.

Read More

Monday, March 27, 2017

To close the socket, don't Close() the socket. Uhmm?

Leave a Comment

I know that TIME_WAIT is an integral part of TCP/IP, but there's many questions on SO (and other places) where multiple sockets are being created per second and the server ends up running out of ephemeral ports.

What I found out is that when using a TCPClient (or Socket for that matter), if I call either the Close() or Dispose() methods the socket's TCP state changes to TIME_WAIT and will respect the timeout period before fully closing.

However, if It just set the variable to null the socket will be fully closed on the next GC run, which can of course be forced, without ever going through a TIME_WAIT state.

This doesn't make a lot of sense for me, since this is an IDisposable object shouldn't the GC also invoke the Dispose() method of the object?

Here's some PowerShell code that demonstrates that (no VS installed on this machine). I used TCPView from Sysinternals to check the sockets state in real time:

$sockets = @() 0..100 | % {     $sockets += New-Object System.Net.Sockets.TcpClient     $sockets[$_].Connect('localhost', 80) }  Start-Sleep -Seconds 10  $sockets = $null  [GC]::Collect() 

Using this method, the sockets never go into a TIME_WAIT state. Same if I just close the app before manually invoking Close() or Dispose()

Can someone shed some light and explain whether this would be a good practice (which I imagine people are going to say it's not).

EDIT

GC's stake in the matter has already been answered, but I am still interested in finding out why this would have any impact on the socket state as this should be controlled by the OS, not .NET.

Also interested in finding out whether it would be good practice to use this method to prevent TIME_WAIT states and ultimately whether this is a bug somewhere (i.e., should all sockets go through a TIME_WAIT state?)

3 Answers

Answers 1

This doesn't make a lot of sense for me, since this is an IDisposable object shouldn't the GC also invoke the Dispose() method of the object?

The Dispose pattern, also known as IDisposable, provides two ways for an unmanaged object to be cleaned up. The Dispose method provides a direct and fast way to clean up the resource. The finalize method, which is called by the garbage collector, is a fail-safe way to make sure that the unmanaged resource is cleaned up in case another developer using the code forgets to call the Dispose method. This is somewhat similar to C++ developers forgetting to call Delete on heap allocated memory - which results in memory leaks.

According to the referenced link:

"Although finalizers are effective in some cleanup scenarios, they have two significant drawbacks:

  1. The finalizer is called when the GC detects that an object is eligible for collection. This happens at some undetermined period of time after the resource is not needed anymore. The delay between when the developer could or would like to release the resource and the time when the resource is actually released by the finalizer might be unacceptable in programs that acquire many scarce resources (resources that can be easily exhausted) or in cases in which resources are costly to keep in use (e.g., large unmanaged memory buffers).

  2. When the CLR needs to call a finalizer, it must postpone collection of the object’s memory until the next round of garbage collection (the finalizers run between collections). This means that the object’s memory (and all objects it refers to) will not be released for a longer period of time."

Using this method, the sockets never go into a TIME_WAIT state. Same if I just close the app before manually invoking Close() or Dispose()

Can someone shed some light and explain whether this would be a good practice (which I imagine people are going to say it's not).

The reason why it is taking a while for it shut down is because the code lingers by default to give the app some time to handle any queued messages. According to the TcpClient.Close method doc on MSDN:

"The Close method marks the instance as disposed and requests that the associated Socket close the TCP connection. Based on the LingerState property, the TCP connection may stay open for some time after the Close method is called when data remains to be sent. There is no notification provided when the underlying connection has completed closing.

Calling this method will eventually result in the close of the associated Socket and will also close the associated NetworkStream that is used to send and receive data if one was created."

This timeout value can be reduced or completely eliminated by the following code:

// Allow 1 second to process queued msgs before closing the socket. LingerOption lingerOption = new LingerOption (true, 1); tcpClient.LingerState = lingerOption; tcpClient.Close();  // Close the socket right away without lingering. LingerOption lingerOption = new LingerOption (true, 0); tcpClient.LingerState = lingerOption; tcpClient.Close(); 

Also interested in finding out whether it would be good practice to use this method to prevent TIME_WAIT states and ultimately whether this is a bug somewhere (i.e., should all sockets go through a TIME_WAIT state?)

As for setting the reference to the TcpClient object to null, the recommended approach is to call the Close method. When the reference is set to null, the GC ends up calling the finalize method. The finalize method eventually calls the Dispose method in order to consolidate the code for cleaning up the unmanaged resource. So, it will work to close the socket - its just not recommended.

In my opinion, it depends on the app whether or not some linger time should be allowed to give the app time to handle queued messages. If I was certain my client app had processed all the necessary messages, then I would probably either give it a linger time of 0 seconds or perhaps 1 second if I thought that might change in the future.

For a very busy client and / or weak hardware - then I might give it more time. For a server, I would have to benchmark different values under load.

Other useful references:

What is the proper way of closing and cleaning up a Socket connection?

Are there any cases when TcpClient.Close or Socket.Close(0) could block my code?

Answers 2

The Socket class has a rather lengthy method protected virtual void Dispose(bool disposing) that is called with true as the parameter from .Dispose() and false as a parameter from the destructor that is called by the garbage collector.

Chances are, your answer to any differences in handling the socket's disposal will be found in this method. Matter of fact, it does not do anything on false from the destructor, so there you have your explanation.

Answers 3

@Bob Bryan posted quite good answer while I was preparing mine. It shows why to avoid finalizers and how to abortively close the connection to avoid TIME_WAITs issue on the server.

I want to refer to a great answer http://stackoverflow.com/a/13088864/2138959 about SO_LINGER to question TCP option SO_LINGER (zero) - when it's required, which might clarify things even more to you and so that you can make you decision in each particular case which approach for closing the socket to use.

To summarize, you should design your client-server communication protocol the way that the client closes the connection to avoid TIME_WAITs on the server.

Read More