Python flask saml throwing saml2.sigver.SigverError Error Message

Has anyone succesfully implemented flask-saml using Windows as dev environment, Python 3.6 and Flask 1.0.2?

I was given the link to the SAML METADATA XML file by our organisation and had it configured on my flask app.

app.config.update({     'SECRET_KEY': 'changethiskeylaterthisisoursecretkey',     'SAML_METADATA_URL': 'https://<url>/FederationMetadata.xml', })  flask_saml.FlaskSAML(app) 

According to the documentation this extension will setup the following routes:

• /saml/logout/: Log out from the application. This is where users go if they click on a “Logout” button.

• /saml/sso/: Log in through SAML.

• /saml/acs/: After /saml/sso/ has sent you to your IdP it sends you back to this path. Also your IdP might provide direct login without needing the /saml/sso/ route.

When I go to one of the routes http://localhost:5000/saml/sso/ I get the error below

saml2.sigver.SigverError saml2.sigver.SigverError: Cannot find ['xmlsec.exe', 'xmlsec1.exe']

I then went to this site https://github.com/mehcode/python-xmlsec/releases/tag/1.3.5 to get xmlsec and install it. However, I'm still getting the same issue.

Here is a screenshot of how I installed xmlsec

where does not seem to find the xmlsec.exe

1 Answers

Answers 1

documentationis asking to have xmlsec1 pre-installed. What you installed is a python binding to xmlsec1.

Get a windows build of xmlsec1 from here or build it from source And make it available in the PATH.

Read More

Long running script from flask endpoint

I've been pulling my hair out trying to figure this one out, hoping someone else has already encountered this and knows how to solve it :)

I'm trying to build a very simple Flask endpoint that just needs to call a long running, blocking php script (think while true {...}). I've tried a few different methods to async launch the script, but the problem is my browser never actually receives the response back, even though the code for generating the response after running the script is executed.

I've tried using both multiprocessing and threading, neither seem to work:

# multiprocessing attempt @app.route('/endpoint') def endpoint():   def worker():     subprocess.Popen('nohup php script.php &', shell=True, preexec_fn=os.setpgrp)    p = multiprocessing.Process(target=worker)   print '111111'   p.start()   print '222222'   return json.dumps({     'success': True   })  # threading attempt @app.route('/endpoint') def endpoint():   def thread_func():     subprocess.Popen('nohup php script.php &', shell=True, preexec_fn=os.setpgrp)    t = threading.Thread(target=thread_func)   print '111111'   t.start()   print '222222'   return json.dumps({     'success': True   }) 

In both scenarios I see the 111111 and 222222, yet my browser still hangs on the response from the endpoint. I've tried p.daemon = True for the process, as well as p.terminate() but no luck. I had hoped launching a script with nohup in a different shell and separate processs/thread would just work, but somehow Flask or uWSGI is impacted by it.

Update

Since this does work locally on my Mac when I start my Flask app directly with python app.py and hit it directly without going through my Nginx proxy and uWSGI, I'm starting to believe it may not be the code itself that is having issues. And because my Nginx just forwards the request to uWSGI, I believe it may possibly be something there that's causing it.

Here is my ini configuration for the domain for uWSGI, which I'm running in emperor mode:

[uwsgi] protocol = uwsgi max-requests = 5000 chmod-socket = 660 master = True vacuum = True enable-threads = True auto-procname = True procname-prefix = michael- chdir = /srv/www/mysite.com module = app callable = app socket = /tmp/mysite.com.sock 

3 Answers

Answers 1

This kind of stuff is the actual and probably main use case for Python Celery (http://www.celeryproject.org/). As a general rule, do not run long running jobs that are CPU-bound in the wsgi process. It's tricky, it's inefficient, and most important thing, it's more complicated than setting up an async task in a celery worker. If you want to just prototype you can set the broker to memory and not using an external server, or run a single threaded redis on the very same machine.

This way you can launch the task, call task.result() which is blocking, but it blocks in an IO-bound fashion, or even better you can just return immediately by retrieving the task_id and build a second endpoint /result?task_id=<task_id> that checks if result is available:

result = AsyncResult(task_id, app=app) if result.state == "SUCCESS":    return result.get() else:    return result.state  # or do something else depending on the state 

This way you have a non-blocking wsgi app that does what is best suited for: short time CPU-unbound calls that have IO calls at most with OS-level scheduling, then you can rely directly to the wsgi server workers|processes|threads or whatever you need to scale the API in whatever wsgi-server like uwsgi, gunicorn, etc. for the 99% of workloads as celery scales horizontally by increasing the number of worker processes.

Answers 2

This approach works for me, it calls the timeout command (sleep 10s) in the command line and lets it work in the background. It returns the response immediately.

@app.route('/endpoint1') def endpoint1():     subprocess.Popen('timeout 10', shell=True)     return 'success1' 

However, not testing on WSGI server, but just locally.

Answers 3

Since this works locally, but doesn't with NGINX and uWSGI ...

• Have you tried adding processes = 2 and threads = 2 to your uWSGI config?
• Have you tried running this without NGINX?

Read More

Docker, Flask, SQLAlchemy: ValueError: invalid literal for int() with base 10: 'None'

I have a flask app that can be initialized successfully and connects to Postgresql database. However, when i try to dockerize this app, i get the below error message. "SQLALCHEMY_DATABASE_URI" is correct and i can connect to it, so i can't figure where I have gone wrong.

docker-compose logs

app_1       |   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line 60, in __init__ app_1       |     self.port = int(port) app_1       | ValueError: invalid literal for int() with base 10: 'None' 

Postgres database connects successfully in Docker container

postgres_1  | LOG:  database system is ready to accept connections 

config.py

from os import environ import os  RDS_USERNAME = environ.get('RDS_USERNAME') RDS_PASSWORD = environ.get('RDS_PASSWORD') RDS_HOSTNAME = environ.get('RDS_HOSTNAME') RDS_PORT = environ.get('RDS_PORT') RDS_DB_NAME = environ.get('RDS_DB_NAME')  SQLALCHEMY_DATABASE_URI = "postgresql+psycopg2://{username}:{password}@{hostname}:{port}/{dbname}"\                           .format(username = RDS_USERNAME, password = RDS_PASSWORD, \                            hostname = RDS_HOSTNAME, port = RDS_PORT, dbname = RDS_DB_NAME) 

flask_app.py (entry point)

def create_app():     app = Flask(__name__, static_folder="./static", template_folder="./static")     app.config.from_pyfile('./app/config.py', silent=True)      register_blueprint(app)     register_extension(app)      with app.app_context():         print(db) -> This prints the correct path for SQLALCHEMY_DATABASE_URI         db.create_all()         db.session.commit()     return app  def register_blueprint(app):     app.register_blueprint(view_blueprint)     app.register_blueprint(race_blueprint)   def register_extension(app):     db.init_app(app)     migrate.init_app(app)   app = create_app()  if __name__ == '__main__':     app.run(host='0.0.0.0', port=8080, debug=True) 

Dockerfile

FROM ubuntu  RUN apt-get update && apt-get -y upgrade  RUN apt-get install -y python-pip && pip install --upgrade pip  RUN mkdir /home/ubuntu  WORKDIR /home/ubuntu/celery-scheduler  ADD requirements.txt /home/ubuntu/celery-scheduler/  RUN pip install -r requirements.txt  COPY . /home/ubuntu/celery-scheduler  EXPOSE 5000  CMD ["python", "flask_app.py", "--host", "0.0.0.0"] 

docker-compose.yml

version: '2'   services:   app:     restart: always     build:        context: .       dockerfile: Dockerfile     volumes:       - .:/app     depends_on:       - postgres    postgres:     restart: always       image: postgres:9.6     environment:       - POSTGRES_USER=${RDS_USERNAME}       - POSTGRES_PASSWORD=${RDS_PASSWORD}       - POSTGRES_HOSTNAME=${RDS_HOSTNAME}       - POSTGRES_DB=${RDS_DB_NAME}     ports:       - "5432:5432" 

1 Answers

Answers 1

You need to set environment variables RDS_USERNAME, RDS_PASSWORD, RDS_HOSTNAME, RDS_PORT , and RDS_DB_NAME in Dockerfile with ENV key value, for example

ENV RDS_PORT 5432 

Read More

Flask + SQLAlchemy: Load database with records by running python script

I'm trying to load my database ONCE with SQLALchemy in a flask app. I thought i could add the records to the database by running a script from the terminal command, but it seems that i'm having difficulties executing the python script?

• Does initializing the app by running export FLASK_APP=app/__init__.py then flask run even loads the database?
• Does starting up the local server each time re

folder structure:

  app     api       __init__.py       log.py     tasks       __init__.py       test.py     __init__.py     models.py     utils.py 

app/api/log.py

from app import app from app.models import Race, db from app.utils  import *   def historical_records():     df_races, df_circuits, constructors, df_drivers, df_results = extract_to_df_race('results', seasons, races_round)     # Check if row exists in table     exists = db.session.query(db.exists().scalar())     if exists is None:         df_races, df_circuits, constructors, df_drivers, df_results = extract_to_df_race('results', seasons, races_round)         save_races_to_db(df_races, db)     else:         print("The database already contains data of 2016 to current race")  def save_races_to_db(df_races, db):     for idx,row in df_races.iterrows():         r = Race()         r.url = df_races.loc[idx,"url"]         r.season = df_races.loc[idx,"season"]         r.raceName = df_races.loc[idx,"raceName"]         db.session.add(r)         try:             db.session.commit()         except Exception as e:             db.session.rollback()             print(str(e))   historical_records() 

I activated the virtual environment, then executed python app/api/log.py but encountered this error:

  File "app/api/log.py", line 1, in <module>     from app import app ImportError: No module named app 

Does initializing the app by running export FLASK_APP=app/__init__.py then flask run even loads the database?

1 Answers

Answers 1

Your issue is that you are using a module inside a package as a script; at that point the top-level module import path is set to the app/api/ directory. At the very least you’d run it as python -m app.api.log to keep the right context.

However, you should instead make your script a Flask command, because that gives you an an active application context.

Make your historical_records() function the command:

import click  from app import app from app.models import Race, db from app.utils  import *   @app.cli.command() def historical_records():     # your function 

Remove the historical_records() call from the end of the module.

You can then run the command with

FLASK_APP=app flask historical_records 

(You don’t need to add /__init__.py to FLASK_APP)

We can’t tell you if flask run will load the database because we can’t see either __init__.py or db.py, nor do I know if you ran the create_all() function.

Read More

Flask-SQLAlchemy Sum of a Column of a Relationship

I do have 2 classes like these =>

class User(db.Model):     __tablename__ = "user"     user_id = db.Column(db.Integer, primary_key=True)     username = db.Column(db.String(32), unique=True, nullable=False)     password = db.Column(db.String(77), unique=False, nullable=False)     server_limit = db.Column(db.Integer, unique=False, nullable=False, server_default="4")     servers = db.relationship('Server', backref='owner', lazy='dynamic')  class Server(db.Model):     __tablename__ = "server"     server_id = db.Column(db.Integer, primary_key=True)     server_admin = db.Column(db.Integer, db.ForeignKey("user.user_id"))     server_port = db.Column(db.Integer, unique=False, nullable=False)     server_slot = db.Column(db.Integer, unique=False, nullable=False, server_default="32") 

Now Im trying to get sum of server_slot column where for example user_id is 1.

I know there is questions with accepted answer about this but the difference is Im trying to do it with servers ( db.relationship ) that I assigned in my User class.

I did it with an alternative method that I created for User class =>

def used(self):     return db.session.execute("SELECT SUM(server.server_slot) FROM server WHERE server_admin={}".format(self.user_id)).scalar() 

How can I do it using db.session.query() ? Im looking for something that I can get it from db.session.query(User).all()

I dont want to use db.session.query(db.func.sum(Server.server_slot)).filter_by(server_admin=self.user_id).scalar() Cause Im passing a list to my Flask page, The list is made by db.session.query(User).all() so I can iterate over it using a for loop inside my Jinja2 Template and show each user information in a list like this =>

{% for user in users %}     <td>user.username</td>     <td>user.server_limit</td>     <td>...</td>     <td>user.used_slots()</td> {% endfor %} 

I can use user.servers.value("server_slot") but it returns only first server's server_slot, I also tried to iterate over user.servers.all() so I could sum their server_slot inside a nested loop, but I can't assign variables any value inside of a loop and get it outside the loop.

Let me know if my question is not clear enough (Cause I know it might be).

1 Answers

Answers 1

Define a hybrid property/expression on your User model.

A simple self-contained example (I've simplified your models):

import random from select import select from flask import Flask from flask_sqlalchemy import SQLAlchemy from sqlalchemy import func from sqlalchemy.ext.hybrid import hybrid_property  app = Flask(__name__)  # Create in-memory database app.config['DATABASE_FILE'] = 'sample_db.sqlite' app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + app.config['DATABASE_FILE'] db = SQLAlchemy(app)   class User(db.Model):     __tablename__ = "user"     user_id = db.Column(db.Integer, primary_key=True)     username = db.Column(db.String(32), unique=True, nullable=False)     servers = db.relationship('Server', backref='owner', lazy='dynamic')      @hybrid_property     def server_slot_count(self):         return sum(server.server_slot for server in self.servers)      @server_slot_count.expression     def server_slot_count(cls):         return (             select([func.sum(Server.server_slot)]).             where(Server.server_admin == cls.user_id).             label('server_slot_count')         )   class Server(db.Model):     __tablename__ = "server"     server_id = db.Column(db.Integer, primary_key=True)     server_admin = db.Column(db.Integer, db.ForeignKey("user.user_id"))     server_slot = db.Column(db.Integer, unique=False, nullable=False, server_default="32")   @app.route('/') def index():     html = []     for user in User.query.all():         html.append('User :{user}; Server Count:{count}'.format(user=user.username, count=user.server_slot_count))      return '<br>'.join(html)   def build_sample_db():     db.drop_all()     db.create_all()      for username in ['DarkSuniuM', 'pjcunningham']:          user = User(             username=username,         )         db.session.add(user)         db.session.commit()         for slot in random.sample(range(1, 100), 5):             server = Server(                 server_admin=user.user_id,                 server_slot=slot             )             db.session.add(server)          db.session.commit()   if __name__ == '__main__':     build_sample_db()     app.run(port=5000, debug=True) 

Your User model now has a property server_slot_count.

{% for user in users %}     <td>user.username</td>     <td>user.server_limit</td>     <td>...</td>     <td>user.server_slot_count</td> {% endfor %} 

Read More

Run Scrapy from Flask

I have this folder structure:

app.py # flask app app/    datafoo/           scrapy.cfg           crawler.py           blogs/                 pipelines.py                  settings.py                 middlewares.py                 items.py                 spiders/                                             allmusic_feed.py                         allmusic_data/                                       delicate_tracks.jl 

scrapy.cfg:

[settings] default = blogs.settings 

allmusic_feed.py:

   class AllMusicDelicateTracks(scrapy.Spider): # one amongst many spiders         name = "allmusic_delicate_tracks"         allowed_domains = ["allmusic.com"]         start_urls = ["http://web.archive.org/web/20160813101056/http://www.allmusic.com/mood/delicate-xa0000000972/songs",                      ]         def parse(self, response):              for sel in response.xpath('//tr'):                 item = AllMusicItem()                 item['artist'] = sel.xpath('.//td[@class="performer"]/a/text()').extract_first()                  item['track'] = sel.xpath('.//td[@class="title"]/a/text()').extract_first()                 yield item 

crawler.py:

from twisted.internet import reactor from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings    def blog_crawler(self, mood):          item, jl = mood  # ITEM = SPIDER         process = CrawlerProcess(get_project_settings())         process.crawl(item, domain='allmusic.com')         process.start()          allmusic = []         allmusic_tracks = []         allmusic_artists = []         try:             # jl is file where crawled data is stored             with open(jl, 'r+') as t:                 for line in t:                     allmusic.append(json.loads(line))         except Exception as e:             print (e, 'try another mood')          for item in allmusic:             allmusic_artists.append(item['artist'])             allmusic_tracks.append(item['track'])         return zip(allmusic_tracks, allmusic_artists) 

app.py :

@app.route('/tracks', methods=['GET','POST']) def tracks(name):     from app.datafoo import crawler      c = crawler()     mood = ['allmusic_delicate_tracks', 'blogs/spiders/allmusic_data/delicate_tracks.jl']     results = c.blog_crawler(mood)     return results 

---

if simply run the app with python app.py, I get the following error:

ValueError: signal only works in main thread 

when I run the app with gunicorn -c gconfig.py app:app --log-level=debug --threads 2, it just hangs there:

127.0.0.1 - - [29/Jan/2018:03:40:36 -0200] "GET /tracks HTTP/1.1" 500 291 "http://127.0.0.1:8080/menu" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" 

lastly, running with gunicorn -c gconfig.py app:app --log-level=debug --threads 2 --error-logfile server.log, I get:

server.log

[2018-01-30 13:41:39 -0200] [4580] [DEBUG] Current configuration:   proxy_protocol: False   worker_connections: 1000   statsd_host: None   max_requests_jitter: 0   post_fork: <function post_fork at 0x1027da848>   errorlog: server.log   enable_stdio_inheritance: False   worker_class: sync   ssl_version: 2   suppress_ragged_eofs: True   syslog: False   syslog_facility: user   when_ready: <function when_ready at 0x1027da9b0>   pre_fork: <function pre_fork at 0x1027da938>   cert_reqs: 0   preload_app: False   keepalive: 5   accesslog: -   group: 20   graceful_timeout: 30   do_handshake_on_connect: False   spew: False   workers: 16   proc_name: None   sendfile: None   pidfile: None   umask: 0   on_reload: <function on_reload at 0x10285c2a8>   pre_exec: <function pre_exec at 0x1027da8c0>   worker_tmp_dir: None   limit_request_fields: 100   pythonpath: None   on_exit: <function on_exit at 0x102861500>   config: gconfig.py   logconfig: None   check_config: False   statsd_prefix:    secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}   reload_engine: auto   proxy_allow_ips: ['127.0.0.1']   pre_request: <function pre_request at 0x10285cde8>   post_request: <function post_request at 0x10285ced8>   forwarded_allow_ips: ['127.0.0.1']   worker_int: <function worker_int at 0x1027daa28>   raw_paste_global_conf: []   threads: 2   max_requests: 0   chdir: /Users/me/Documents/Code/Apps/app   daemon: False   user: 501   limit_request_line: 4094   access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"   certfile: None   on_starting: <function on_starting at 0x10285c140>   post_worker_init: <function post_worker_init at 0x10285c848>   child_exit: <function child_exit at 0x1028610c8>   worker_exit: <function worker_exit at 0x102861230>   paste: None   default_proc_name: app:app   syslog_addr: unix:///var/run/syslog   syslog_prefix: None   ciphers: TLSv1   worker_abort: <function worker_abort at 0x1027daaa0>   loglevel: debug   bind: ['127.0.0.1:8080']   raw_env: []   initgroups: False   capture_output: False   reload: False   limit_request_field_size: 8190   nworkers_changed: <function nworkers_changed at 0x102861398>   timeout: 120   keyfile: None   ca_certs: None   tmp_upload_dir: None   backlog: 2048   logger_class: gunicorn.glogging.Logger [2018-01-30 13:41:39 -0200] [4580] [INFO] Starting gunicorn 19.7.1 [2018-01-30 13:41:39 -0200] [4580] [DEBUG] Arbiter booted [2018-01-30 13:41:39 -0200] [4580] [INFO] Listening at: http://127.0.0.1:8080 (4580) [2018-01-30 13:41:39 -0200] [4580] [INFO] Using worker: threads [2018-01-30 13:41:39 -0200] [4580] [INFO] Server is ready. Spawning workers [2018-01-30 13:41:39 -0200] [4583] [INFO] Booting worker with pid: 4583 [2018-01-30 13:41:39 -0200] [4583] [INFO] Worker spawned (pid: 4583) [2018-01-30 13:41:39 -0200] [4584] [INFO] Booting worker with pid: 4584 [2018-01-30 13:41:39 -0200] [4584] [INFO] Worker spawned (pid: 4584) [2018-01-30 13:41:39 -0200] [4585] [INFO] Booting worker with pid: 4585 [2018-01-30 13:41:39 -0200] [4585] [INFO] Worker spawned (pid: 4585) [2018-01-30 13:41:40 -0200] [4586] [INFO] Booting worker with pid: 4586 [2018-01-30 13:41:40 -0200] [4586] [INFO] Worker spawned (pid: 4586) [2018-01-30 13:41:40 -0200] [4587] [INFO] Booting worker with pid: 4587 [2018-01-30 13:41:40 -0200] [4587] [INFO] Worker spawned (pid: 4587) [2018-01-30 13:41:40 -0200] [4588] [INFO] Booting worker with pid: 4588 [2018-01-30 13:41:40 -0200] [4588] [INFO] Worker spawned (pid: 4588) [2018-01-30 13:41:40 -0200] [4589] [INFO] Booting worker with pid: 4589 [2018-01-30 13:41:40 -0200] [4589] [INFO] Worker spawned (pid: 4589) [2018-01-30 13:41:40 -0200] [4590] [INFO] Booting worker with pid: 4590 [2018-01-30 13:41:40 -0200] [4590] [INFO] Worker spawned (pid: 4590) [2018-01-30 13:41:40 -0200] [4591] [INFO] Booting worker with pid: 4591 [2018-01-30 13:41:40 -0200] [4591] [INFO] Worker spawned (pid: 4591) [2018-01-30 13:41:40 -0200] [4592] [INFO] Booting worker with pid: 4592 [2018-01-30 13:41:40 -0200] [4592] [INFO] Worker spawned (pid: 4592) [2018-01-30 13:41:40 -0200] [4595] [INFO] Booting worker with pid: 4595 [2018-01-30 13:41:40 -0200] [4595] [INFO] Worker spawned (pid: 4595) [2018-01-30 13:41:40 -0200] [4596] [INFO] Booting worker with pid: 4596 [2018-01-30 13:41:40 -0200] [4596] [INFO] Worker spawned (pid: 4596) [2018-01-30 13:41:40 -0200] [4597] [INFO] Booting worker with pid: 4597 [2018-01-30 13:41:40 -0200] [4597] [INFO] Worker spawned (pid: 4597) [2018-01-30 13:41:40 -0200] [4598] [INFO] Booting worker with pid: 4598 [2018-01-30 13:41:40 -0200] [4598] [INFO] Worker spawned (pid: 4598) [2018-01-30 13:41:40 -0200] [4599] [INFO] Booting worker with pid: 4599 [2018-01-30 13:41:40 -0200] [4599] [INFO] Worker spawned (pid: 4599) [2018-01-30 13:41:40 -0200] [4600] [INFO] Booting worker with pid: 4600 [2018-01-30 13:41:40 -0200] [4600] [INFO] Worker spawned (pid: 4600) [2018-01-30 13:41:40 -0200] [4580] [DEBUG] 16 workers [2018-01-30 13:41:47 -0200] [4583] [DEBUG] GET /menu [2018-01-30 13:41:54 -0200] [4584] [DEBUG] GET /tracks 

---

NOTE:

in this SO answer I've learned that in order to integrate Flask and Scrapy you can either use:

1. Python subprocess

2. Twisted-Klein + Scrapy

3. ScrapyRT

but I haven't had any luck adapting my specific code to these solutions.

I reckon a subprocess would be simpler and suffice, because user experience rarely requires a scraping thread, but am not sure.

could anyone please point me in the right direction here?

1 Answers

Answers 1

Here's a minimal example how you can do it with ScrapyRT.

This is the project structure:

project/ ├── scraping │   ├── example │   │   ├── __init__.py │   │   ├── items.py │   │   ├── middlewares.py │   │   ├── pipelines.py │   │   ├── settings.py │   │   └── spiders │   │       ├── __init__.py │   │       └── quotes.py │   └── scrapy.cfg └── webapp     └── example.py 

scraping directory contains the Scrapy project. This project contains one spider quotes.py to scrape some quotes from quotes.toscrape.com:

# -*- coding: utf-8 -*- from __future__ import unicode_literals  import scrapy   class QuotesSpider(scrapy.Spider):     name = 'quotes'     start_urls = ['http://quotes.toscrape.com/']      def parse(self, response):         for quote in response.xpath('//div[@class="quote"]'):             yield {                 'author': quote.xpath('.//small[@class="author"]/text()').extract_first(),                 'text': quote.xpath('normalize-space(./span[@class="text"])').extract_first()             } 

In order to start ScrapyRT and listen to requests for scraping, go to the Scrapy project's directory scraping and issue scrapyrt command:

$ cd ./project/scraping $ scrapyrt 

ScrapyRT will now listen on localhost:9080.

webapp directory contains simple Flask app that scrapes quotes on demand (using the spider above) and simply displays them to user:

from __future__ import unicode_literals  import json import requests  from flask import Flask  app = Flask(__name__)  @app.route('/') def show_quotes():     params = {         'spider_name': 'quotes',         'start_requests': True     }     response = requests.get('http://localhost:9080/crawl.json', params)     data = json.loads(response.text)     result = '\n'.join('<p><b>{}</b> - {}</p>'.format(item['author'], item['text'])                        for item in data['items'])     return result 

To start the app:

$ cd ./project/webapp $ FLASK_APP=example.py flask run 

Now when you point the browser on localhost:5000, you'll the list of quotes freshly scraped from quotes.toscrape.com.

Read More

How to properly close mysql connections in sqlalchemy?

I would like to know what is proper way to close all mysql connections in sqlalchemy. For the context, it is a Flask application and all the views share the same session object.

engine = create_engine("mysql+pymysql://root:root@127.0.0.1/my_database")  make_session = sessionmaker(bind=engine, autocommit=False)  session = ScopedSession(make_session)() 

And when the app is teared down, the session is closed and engine is disposed

session.close() engine.dispose() 

But according to database log, I still have a lot of errors like [Warning] Aborted connection 940 to db: 'master' user: 'root' host: '172.19.0.7' (Got an error reading communication packets).

I have tried some solutions, including calling gc.collect() and engine.pool.dispose() but without success ...

I suspect there are still some connections opened by the engine behind the scene and they need to be closed. Is there anyway to list all the sessions/connections opened by the engine?

After spending a lot of time on this, any advice/help/pointer will be much appreciated! Thanks.

P.S: the dispose and close calls are inspired from How to close sqlalchemy connection in MySQL. Btw, what is a 'checked out' connection?

1 Answers

Answers 1

This may not answer your question completely, but I've been using this method to make sure all my sessions are closed. Every function that uses a session gets the provide_session decorator. Note the session=None argument must be present.

e.g

@provide_session() def my_func(session=None):     do some stuff     session.commit() 

I saw it used in the Incubator-Airflow project and really liked it.

import contextlib from functools import wraps  ... Session = ScopedSession(make_session)  @contextlib.contextmanager def create_session():     """     Contextmanager that will create and teardown a session.     """     session = Session()     try:         yield session         session.expunge_all()         session.commit()     except:         session.rollback()         raise     finally:         session.close()   def provide_session(func):     """     Function decorator that provides a session if it isn't provided.     If you want to reuse a session or run the function as part of a     database transaction, you pass it to the function, if not this wrapper     will create one and close it for you.     """     @wraps(func)     def wrapper(*args, **kwargs):         arg_session = 'session'          func_params = func.__code__.co_varnames         session_in_args = arg_session in func_params and \             func_params.index(arg_session) < len(args)         session_in_kwargs = arg_session in kwargs          if session_in_kwargs or session_in_args:             return func(*args, **kwargs)         else:             with create_session() as session:                 kwargs[arg_session] = session                 return func(*args, **kwargs)      return wrapper 

Read More

Apache Flask Error 13, permission denied

I deployed a Flask application using Apache on AWS for the first time. The HTML pages load, however things like uploading files, writing files and reading files do not seem to work. In the below example I'm calling this specific function that writes data received from a URL. But here, I have disabled that, and the code merely has to read the file that is already there. So, export_po_list.xml is already there, and I have checked this from the terminal. This same code runs just fine locally in my PC.

Checking /var/log/apache2/error.log reveals

IOError: [Errno 13] Permission denied: 'export_po_list.xml' 

I did chmod 777 -R to the whole folder that has this flask application. It still doesn't work.

 def po_data(a,b,c):      array0 = []     array1 = []     array2 = []     array3 = []     array4 = []     array5 = []     array6 = []     array7 = []     array8 = []     array9 = []     array10 = []     array11 = []     array12 = []     array13 = []     array14 = []     array15 = []     array16 = []     array17 = []     array18 = []     array19 = []     array20 = []      url_begin = "https://34.239.8.24:44300/sap/opu/odata/sap/ZRECEASY_ALL_OPEN_PO_SRV/ZRECEASY_ALL_OPEN_POSet?$filter=ImBstyp eq '"     url_mid_1 = "' and ImBsart eq '"     url_mid_2 = "' and ImErnam eq '"     url_end = "'"     final_url = url_begin + a + url_mid_1 + b + url_mid_2 + c + url_end     print "\n\n"     print final_url     print "\n\n"     auth_get_po_data ='S4H_FIN','Welcome1'     headers_get_po_data = {"Content-type":'application/json;charset=utf-8'}     final_url = "https://34.239.8.24:44300/sap/opu/odata/sap/ZRECEASY_ALL_OPEN_PO_SRV/ZRECEASY_ALL_OPEN_POSet?$filter=ImBstyp eq 'F' and ImBsart eq 'NB' and ImErnam eq 'S4H_MM'"      #Post data back     # final_url = "https://34.239.8.24:44300/sap/opu/odata/sap/ZRECEASY_ALL_OPEN_PO_SRV/ZRECEASY_ALL_OPEN_POSet?    # r_get_po_data = requests.get(final_url,headers=headers_get_po_data,auth=auth_get_po_data, verify=False)    # print r_get_po_data.text     print os.getcwd()  # Write temporary XML file to work on parsing #   file = open('export_po_list.xml', 'w') #   file.write(r_get_po_data.text) #   file.close()  # Read XML file     print os.getcwd()     tree = ET.parse('export_po_list.xml')     root = tree.getroot()  #Extract relevant info     for child in root:         for child2 in child:             for child3 in child2:                 counter = 1                 for child4 in child3:                     # 5 24                     if (counter == 5):                         array0.append(str(child4.text))                     elif (counter == 6):                         array1.append(str(child4.text))                     elif (counter == 7):                         array2.append(str(child4.text))                     elif (counter == 8):                         array3.append(str(child4.text))                     elif (counter == 9):                         array4.append(str(child4.text))                     elif (counter == 10):                         array5.append(str(child4.text))                     elif (counter == 11):                         array6.append(str(child4.text))                     elif (counter == 12):                         array7.append(str(child4.text))                     elif (counter == 13):                         array8.append(str(child4.text))                     elif (counter == 14):                         array9.append(str(child4.text))                     elif (counter == 15):                         array10.append(str(child4.text))                     elif (counter == 16):                         array11.append(str(child4.text))                     elif (counter == 17):                         array12.append(str(child4.text))                     elif (counter == 18):                         array13.append(str(child4.text))                     elif (counter == 19):                         array14.append(str(child4.text))                     elif (counter == 20):                         array15.append(str(child4.text))                     elif (counter == 21):                         array16.append(str(child4.text))                     elif (counter == 22):                         array17.append(str(child4.text))                     elif (counter == 23):                         array18.append(str(child4.text))                     elif (counter == 24):                         array19.append(str(child4.text))                     elif (counter == 25):                         array20.append(str(child4.text))                     counter = counter + 1      return array0, array1, array2, array3, array4, array5, array6, array7, array8, array9, array10, array11, array12, array13, array14, array15, array16, array17, array18, array19, array20 

1 Answers

Answers 1

Don't use a relative path name for the file, you need to calculate an absolute path and ensure the location is a writable directory. This is necessary as the current working directory for Apache is usually the root directory, which isn't writable to the user your code runs as.

For more details see the mod_wsgi documentation at:

• http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#application-working-directory
• http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#access-rights-of-apache-user

Read More

Flask Form data duplicates on submit

i am trying to populate a table of current values, then change it with intention of finding the diffs of original and after. i simplify my code in the following to replicate the issue: -

webapp.py

from flask import Flask, render_template from flask_wtf import FlaskForm from wtforms import StringField, DecimalField, fields import pandas as pd  app=Flask(__name__) app.config['SECRET_KEY'] = 'wtf'  class stockForm(FlaskForm):     stock=StringField()     price= DecimalField()      def __init__(self, csrf_enabled=False, *args, **kwargs):         super(stockForm, self).__init__(csrf_enabled=csrf_enabled, *args, **kwargs)  class stockListForm(FlaskForm):     stockItem=fields.FieldList(fields.FormField(stockForm))   @app.route('/sEntry', methods=['GET','POST']) def sEntry():     form=stockListForm()     stocklist=pd.DataFrame(data=[['abc',10.17],['bcd',11.53],['edf',12.19]],columns=['stock','price'])          for stock in stocklist.itertuples():         sForm=stockForm()         sForm.stock=stock.stock         sForm.price=stock.price         form.stockItem.append_entry(sForm)      if form.validate_on_submit():         results = []         for idx, data in enumerate(form.stockItem.data):             results.append(data)         print(results)         del form         return render_template('results.html', results=results)     print(form.errors)     return render_template('sEntry.html',form=form)   if __name__=='__main__':     app.run(debug=True, use_reloader=True, host='0.0.0.0', port=int('5050')) 

sEntry.html

<html lang="en">   <head>     <meta charset="utf-8">   </head> <body>      <form action="" method="POST" name="form">     {{ form.name}}     {{ form.hidden_tag() }}     <div>         <table>             <thead >             <tr class="col">                 <th style="width: 30px">stock</th>                 <th style="width: 50px">price</th>             </tr>             </thead>             {% for stock in form.stockItem %}             <tr class="col">                 <td>{{ stock.stock }}</td>                 <td>{{ stock.price }}</td>             </tr>             {% endfor %}         </table>     </div>     <p><input type="submit" name="edit" value="Send"></p>     </form> </body> </html> 

results.html

<ul> {% for line in results %}     <li>{{ line }}</li> {% endfor %}  </ul> 

if I am to change the values of a few of the field, the variable results generated will have duplicate of 6 rows of data from my original 3 rows in the dataframes e.g.

{'price': Decimal('10.17'), 'stock': 'abc'} {'price': Decimal('13'),    'stock': 'bcd'} {'price': Decimal('12.19'), 'stock': 'edf'} {'price': 10.17, 'stock': 'abc'} {'price': 11.529999999999999, 'stock': 'bcd'} {'price': 12.19, 'stock': 'edf'} 

Furthermore, i also have issues my original Decimals used in turning into some long float values, in above example, i change bcd value from 11.53 to 13, the original value become long float figure, the rest that i didnt edit stay as original.

I could have the dirty solution of cutting the results into half and compare values between both halves, rounding those long floats to find values that have change, but seems very inefficient.

can anyone assist?

1 Answers

Answers 1

Firstly, you need to use proper Decimal type in the Pandas DataFrame. (Which can be handled by Pandas by using Numpy's dtype with an object).

Secondly, you were filling the form with original data when POST request occurred.

Somewhat fixed view function would look like this:

@app.route('/', methods=['GET','POST']) def sEntry():     # Create form and fill it with request data     form = stockListForm(request.form)      # Set up initial data with proper Decimal objects     stocklist=pd.DataFrame(data=[['abc',Decimal('10.17')],['bcd',Decimal('11.53')],['edf',Decimal('12.19')]],columns=['stock','price'])          # Handle valid POST request     if form.validate_on_submit():         # Convert form data to dictionary (so we can later easily query stock price)         stocks = {i['stock']: i['price'] for i in form.stockItem.data}          # Generate result (as generator) ...         results = ((i.stock, i.price, i.price - stocks[i.stock]) for i in stocklist.itertuples())          # ... and push it to template         return render_template('results.html', results=results)      print(form.errors)      # ...build initial form for GET request      for stock in stocklist.itertuples():         sForm=stockForm()         sForm.stock=stock.stock         sForm.price=stock.price         form.stockItem.append_entry(sForm)      return render_template('sEntry.html',form=form) 

Read More

Dynamic database connection Flask-SQLAlchemy

i need to connect two database. the default database is fixed but the other one is dynamic, its based on URL.

for example if url is : yourapp.myweb.com then second database name will be yourapp

i try connect database into init.py but its show me following error

builtins.AssertionError AssertionError: A setup function was called after the first request was handled.  This usually indicates a bug in the application where a module was not imported and decorators or other functionality was called too late. To fix this make sure to import all your view modules, database models and everything related at a central place before the application starts serving requests. 

here is my init.py

from flask import Flask,session from flask_sqlalchemy import SQLAlchemy import os app = Flask(__name__,static_url_path='/static')  #  Database Connection database = request.url.split("/")[2].split(".")[0] app.config['SQLALCHEMY_DATABASE_URI'] = "mysql+pymysql://root:root@localhost/main_database" app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True app.config['SQLALCHEMY_BINDS'] = {     'user_db': 'mysql+pymysql://root:root@localhost/database_'+str(database), #dynamic Connection } db = SQLAlchemy(app) db.create_all() db.create_all(bind=['user_db']) # db.init_app(app)  from . import views 

here is the viwe.py

@app.route('/login', methods = ['GET']) def index():     try:         from .model import Users         # Some Code     except Exception as e:         raise e         # return "Failed to login ! Please try again." 

here is the model.py

from application import db class Users(db.Model):     __bind_key__ = 'user_db'     __tablename__ = 'users'     id = db.Column(db.Integer, primary_key = True)     email = db.Column(db.String(50))     name = db.Column(db.String(50))     password = db.Column(db.String())      def __repr__(self):         return '<User %r>' % self.name 

1 Answers

Answers 1

As I said in one of my comments, this might be a problem with the database connection. Here's what I'd check for:

1. First of all, make sure you have the right engine installed in your virtual environment (you can check easily by running pip list; just in case, let me insist that libraries need to be installed in a virtual environment). Make sure you have not pymysql, but the port to Python3, called mysqlclient. pymysql only works with Python2. In order to install this library, you need to install first the Python and MySQL development headers. For example, in Debian/Ubuntu:

   sudo apt-get install python-dev libmysqlclient-dev 

   Then you can install the library with the following command:

   pip install mysqlclient 

2. If this is installed, make sure you can actually connect to the database using the library. Open a Python shell within the virtual environment and type the following (from the example in github):

   import pymysql.cursors  connection = pymysql.connect(host='<you_host>',                              user='<user>',                              password='<password>',                              db='<database_name>',                              charset='utf8mb4',                              cursorclass=pymysql.cursors.DictCursor)  try:     with connection.cursor() as cursor:         do_something() except:     pass 

3. If this works, make sure you're running the most recent version of Flask (0.12 at the moment; this again you can check by running pip list), as there are several bugs related to running Flask in DEBUG mode that have been fixed over time.

4. It's surely not the case here, but another sanity check is verifying that no other process is running on the port that you want to use for Flask.

If all of the above is working fine, I'd need to see a bit of the stack trace to figure out what is actually going on.

Read More

Is it safe to store my 'next' url in a signed cookie and redirect to it carefree?

I'm using Flask and it's occurred to me it could be a rather elegant solution to redirect back to the user's last page after login/logout by simply placing a session['next'] = request.url at each endpoint of my application and to just have my login/logout functions redirect right to session.get('next'). This is even similar to an option in the Flask-Login extension if you enable USE_SESSION_FOR_NEXT.

I would like to confirm this is a safe workflow but am not security-savvy to recognize if there are any ways to spoof the request.url or if I should still be validating the next url prior to redirecting, as is specified here:

http://flask.pocoo.org/snippets/62/

Is there a reason this method is not more commonly deployed? It seems like a nice, clean, easy solution that keeps URL's clean, minimizes for fields/processing, and removes a vulnerability to open redirect attacks if you are not taking the extra steps to validate the next url. What's the catch?

4 Answers

Answers 1

TL;DR

1. Yes, it is safe.
2. Don't worry about hiding the url, or preventing "spoofing". In reality, you can do neither, so you prepare for both.
3. Potential bug: once you set the redirect on the session, the next login will follow it no matter what, until the session expires. A way to remedy this is just save the redirect path as a GET parameter instead (see bottom of answer).
4. Here is a nice cheat sheet for redirect and forwarding security concerns.

This is a safe workflow, assuming your server authenticates incoming requests to all secure resources.

When you ask "if I should still be validating the next url prior to redirecting", the answer to that is no, but you need to validate all requests to that url (after redirecting) to make sure they are logged in.

In your question, it sounds like you are trying to keep the url hidden from the user until they are logged in. That shouldn't be necessary.

For example, let's say you have two urls:

url 1:  "/login"  # anyone can access this url 2:  "/secure_page"  # only a logged in user can access this 

Let's say a user is not logged in, and tries to navigate to yoursite.com/secure_page. They should be redirected to your login page every time. If they know the secure_page url, that shouldn't compromise your security at all. In fact, they must already know that url because they navigated to that page in the first place, so either they typed it in, clicked a link, or it was saved in a bookmark or a browsing history. The important thing is that when you handle requests to secure_page, you require them to be logged in.

You ask if they can "spoof" the url that they are redirected to. They cannot, when you save it in the session like that. However, they can hit any url they want after they log in, so it doesn't matter if they "spoof" that url.

So since they already know that url and they can hit any url they want after they log in, you don't need to keep that url secret from the user. The only advantage of doing that is an aesthetically cleaner url. This is why you see many login pages that look like this:

http://yoursite.com/login?next=secure_page 

What is happening there is they are saving that next url as an HTTP GET parameter. While it is less "clean", it is more explicit, so it has its pros and cons. And if they were to revisit the login page later, that redirect would no longer occur, which may be your desired behavior. With your current code, once that redirect is on the session, this next login after that will follow the redirect until the session expires, which could be after the user walks away and someone else uses that web browser.

Many sites do it the way I showed above. Django does it that way. In this case, a malicious link could provide the "next" url and have it redirect to some phishing site, but that could be easily prevented by ensuring the "next" url doesn't navigate away from your domain.

Here is a basic cheat sheet for redirect/forward security (you are redirecting, but others landing here may be considering forwarding requests).

Answers 2

To be honest, I can't think of any practical way to exploit this. Rather, I can't think of any practical way to exploit this, assuming you are properly validating incoming requests on other pages, and have properly implemented authentication and access control. That being said, I know nothing about your application, and there are certainly attackers out there a lot brighter than me, so take that with a grain of salt.

That being said, I don't see any reason why you shouldn't validate the URL, at the very least to make sure you're redirecting to a page on the same domain. The performance cost is trivial, and I'm of the opinion that anything related to the login/logout flow deserves an extra level of scrutiny. I would also make sure that you have an acceptable default case for when you receive a request without this field in the cookie.

Answers 3

As @brendan commented, this solution will work fine, use a @login_required decorator to protect the view.

Two cases:

Loggin in

You are at /index (non-protected view) >> login successfully to a @login_protected view, in this case, if you go back, as /index is accessible for everybody, you really do not care

Logging out

You are at /profile view (protected view) >> successful logout to a non-protected view, if you go back, the decorator will not allow you to enter the view

Answers 4

There is one small issue though. If you have multiple frames in your html there is an exploit and is called as clickjacking. You will need to set the following headers so that browser takes care of the remaining.

<X-Frame-Options','DENY'> 

Though it does not concern redirection but it can be used here. Be careful.

Read More

flask-dance: multiple auth-providers

I have successfully followed the examples in the documentation of Flask-Dance to add Oauth from GitHub, and manage the users with SQLAlchemy. However, I am unable to add more providers. I tried to simply add another blueprint for Twitter and registering it, but I get various errors when trying to login with Twitter. Also, I end up with lots of duplicated code, so this is not a good approach.

Is there a better way to add another provider?

import sys from flask import Flask, redirect, url_for, flash, render_template from flask_sqlalchemy import SQLAlchemy from sqlalchemy.orm.exc import NoResultFound from flask_dance.contrib.github import make_github_blueprint, github from flask_dance.consumer.backend.sqla import OAuthConsumerMixin, SQLAlchemyBackend from flask_dance.consumer import oauth_authorized, oauth_error from flask_login import (     LoginManager, UserMixin, current_user,     login_required, login_user, logout_user )  # setup Flask application app = Flask(__name__) app.secret_key = "supersekrit" blueprint = make_github_blueprint(     client_id="my-key-here",     client_secret="my-secret-here", ) app.register_blueprint(blueprint, url_prefix="/login")  # setup database models app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///multi.db" db = SQLAlchemy()  class User(db.Model, UserMixin):     id = db.Column(db.Integer, primary_key=True)     username = db.Column(db.String(256), unique=True)     # ... other columns as needed  class OAuth(db.Model, OAuthConsumerMixin):     user_id = db.Column(db.Integer, db.ForeignKey(User.id))     user = db.relationship(User)  # setup login manager login_manager = LoginManager() login_manager.login_view = 'github.login'  @login_manager.user_loader def load_user(user_id):     return User.query.get(int(user_id))  # setup SQLAlchemy backend blueprint.backend = SQLAlchemyBackend(OAuth, db.session, user=current_user)  # create/login local user on successful OAuth login @oauth_authorized.connect_via(blueprint) def github_logged_in(blueprint, token):     if not token:         flash("Failed to log in with {name}".format(name=blueprint.name))         return     # figure out who the user is     resp = blueprint.session.get("/user")     if resp.ok:         username = resp.json()["login"]         query = User.query.filter_by(username=username)         try:             user = query.one()         except NoResultFound:             # create a user             user = User(username=username)             db.session.add(user)             db.session.commit()         login_user(user)         flash("Successfully signed in with GitHub")     else:         msg = "Failed to fetch user info from {name}".format(name=blueprint.name)         flash(msg, category="error")  # notify on OAuth provider error @oauth_error.connect_via(blueprint) def github_error(blueprint, error, error_description=None, error_uri=None):     msg = (         "OAuth error from {name}! "         "error={error} description={description} uri={uri}"     ).format(         name=blueprint.name,         error=error,         description=error_description,         uri=error_uri,     )     flash(msg, category="error")  @app.route("/logout") @login_required def logout():     logout_user()     flash("You have logged out")     return redirect(url_for("index"))  @app.route("/") def index():     return render_template("home.html")  # hook up extensions to app db.init_app(app) login_manager.init_app(app)  if __name__ == "__main__":     if "--setup" in sys.argv:         with app.app_context():             db.create_all()             db.session.commit()             print("Database tables created")     else:         app.run(debug=True) 

0 Answers

Read More

uwsgi: *** no app loaded. going in full dynamic mode ***

wapp.py is in:

/var/www/KRAKEN/public/site.me/wapp 

with the structure:

|-- __init__.py |-- wapp.py |-- wapp.ini |-- mod_db |-- mod_form |-- multimedia |-- pip-selfcheck.json |-- pyvenv.cfg |-- requirements.txt |-- run_now.py |-- static `-- templates 

My wapp.ini which i run in emperor mode is:

[uwsgi] dir             = /var/www/KRAKEN/public/site.me chdir           = %(dir) master          = true processes       = 1 socket          = /run/uwsgi/wapp.sock chmod-socket    = 666 enable-threads  = true vacuum          = true virtualenv      = %(dir) binary-path     = %(dir)/bin/uwsgi wsgi-file       = wapp.py mount           = /%dir/wapp/app=wapp.py logto           = %(dir)/uwsgi.log 

Previously the app was on the website root /var/www/KRAKEN/public/site.me, but i dont know what's going on because i updated all the pertinent config parameters, but obviously i'm missing something. I can run the app with no errors, inside the virtual env with: python wapp.py

Any tips? Thanks!

1 Answers

Answers 1

I used my old config, changing just the virtualenv parameter and bin, worked. I bet i changed a lot of things too fast and one thing broke the other in a spiral of doom.

Read More

Flask - Active Directory Authentication

I made a small Flask application and I would like users to be able to authenticate with their Windows NT IDs. I am not a part of the IT team, so I have limited insight into this area and my IT team is not experienced with Python.

How easy would it be to configure this? I tried to do some Googling and I saw LDAP modules and Flask-Security. I am hoping for a quick guide or to be pointed into a specific direction.

• There is an existing Active Directory and a lot of our internal websites use NT authentication
• I made a Flask app that I will be porting to our internal network
• I want users to be able to login to the site with their NT ID
• I need to know what information I need (an LDAP server and port?) or what I need to do with IT to get this configured properly without breaking any security protocols

Thanks!

1 Answers

Answers 1

It is quite easy to work with Flask as it is lightweight and plugin based Python web framework

Things you will need for LDAP Configuration

• LDAP Host
• LDAP Domain
• LDAP Profile Key

You need to install Flask-LDAP plugin

pip install Flask-LDAP 

and here is a basic example to get you started:

from flask import Flask from flask.ext.ldap import LDAP, login_required  app = Flask(__name__) app.debug = True  app.config['LDAP_HOST'] = 'ldap.example.com' app.config['LDAP_DOMAIN'] = 'example.com' app.config['LDAP_SEARCH_BASE'] = 'OU=Domain Users,DC=example,DC=com'  ldap = LDAP(app) app.secret_key = "welfhwdlhwdlfhwelfhwlehfwlehfelwehflwefwlehflwefhlwefhlewjfhwelfjhweflhweflhwel" app.add_url_rule('/login', 'login', ldap.login, methods=['GET', 'POST'])  @app.route('/') @ldap.login_required def index():         pass  # @app.route('/login', methods=['GET', 'POST']) # def login(): #     pass  if __name__ == '__main__': app.run(debug=True, host="0.0.0.0") 

More details can be found here

Read More

Flask application GET returning the same thing twice

I currently have two methods to which I call simultaneously (via HTTP in Java)

For some reason, there is an instance in which getAcc() returns the same account info twice? I don't quite understand why this is.

I think its possible that the second call to get_account is happening before toggleUse() is called (and therefore the IN_USE variable is not set to 1 yet). Does anyone know how to fix this? I've done some reading on the web and I believe the term is Serialization. I've seen this mostly in terms of databases, but have seen almost no references on how to "lock" the method. I could create a lock and do "with lock:" but Ive heard that's not the way to go.

@app.route('/getAcc') def get_account():     try:         cursor.execute("SELECT USER,PASS FROM ACCOUNTS WHERE TUT_DONE = 0 AND IN_USE = 0 LIMIT 1;")         return jsonify(data=cursor.fetchall())     except (AttributeError, MySQLdb.OperationalError):         open_db()         return get_account()   @app.route('/toggleUse', methods=['POST']) def toggle_use():     try:         username = request.values['username']         update_stmt = (           "UPDATE ACCOUNTS SET IN_USE = !IN_USE WHERE USER = (%s)"         )         data = (username,)         cursor.execute(update_stmt,data)         db.commit()         return 'Account ' + str(username) +  ' IN_USE toggled'      except (AttributeError, MySQLdb.OperationalError):         open_db()         return toggle_use() 

1 Answers

Answers 1

I would have to test it, but suspect you have an issue with your except opening a db connection and calling itself.

Have you tried to create a connection that give you a pool you can just call?

connection = pyodbc.connect(connection_string, autocommit=True) #I think you can set autocommit here.  @app.route('/getAcc') def get_account():     try:         cursor = connection.cursor()         cursor.execute("SELECT USER,PASS FROM ACCOUNTS WHERE TUT_DONE = 0 AND IN_USE = 0 LIMIT 1;")         return jsonify(data=cursor.fetchall())     except (AttributeError, MySQLdb.OperationalError):         # Return a meaningful message 

Same thing for the other function.

Read More

Socket.io POST Request Logs

I am running socket.io on an Apache server through Python Flask. We're integrating it into an iOS app and we're having a weird issue.

From the client side code in the app (written in Swift), I can view the actual connection log and see the connection established with the client's IP and the requests being made. The client never receives the information back (or any information back; even when using a global event response handler) from the socket server.

I wrote a very simple test script in Javascript on an HTML page and sent requests that way and received the proper responses back. With that said, it seems to likely be an issue with iOS. I've found these articles (but none of them helped fix the problem):

https://github.com/nuclearace/Socket.IO-Client-Swift/issues/95 https://github.com/socketio/socket.io-client-swift/issues/359

My next thought is to extend the logging of socket.io to find out exact what data is being POSTed to the socket namespace. Is there a way to log exactly what data is coming into the server (bear in mind that the 'on' hook on the server side that I've set up is not getting any data; I've tried to log it from there but it doesn't appear to even get that far).

I found mod_dumpio for Linux to log all POST requests but I'm not sure how well it will play with multi-threading and a socket server.

Any ideas on how to get the exact data being posted so we can at least troubleshoot the syntax and make sure the data isn't being malformed when it's sent to the server?

Thanks!

2 Answers

Answers 1

I assume you have verified that Apache does get the POST requests. That should be your first test, if Apache does not log the POST requests coming from iOS, then you have a different kind of problem.

If you do get the POST requests, then you can add some custom code in the middleware used by Flask-SocketIO and print the request data forwarded by Apache's mod_wsgi. The this is in file flask_socketio/init.py. The relevant portion is this:

class _SocketIOMiddleware(socketio.Middleware):      # ...      def __call__(self, environ, start_response):         # log what you need from environ here         environ['flask.app'] = self.flask_app         return super(_SocketIOMiddleware, self).__call__(environ, start_response) 

You can find out what's in environ in the WSGI specification. In particular, the body of the request is available in environ['wsgi.input'], which is a file-like object you read from.

Keep in mind that once you read the payload, this file will be consumed, so the WSGI server will not be able to read from it again. Seeking the file back to the position it was before the read may work on some WSGI implementations. A safer hack I've seen people do to avoid this problem is to read the whole payload into a buffer, then replace environ['wsgi.input'] with a brand new StringIO or BytesIO object.

Answers 2

Are you using flask-socketio on the server side? If you are, there is a lot of debugging available in the constructor.

socketio = SocketIO(app, async_mode=async_mode, logger=True, engineio_logger=True)

Read More

How to send 1 time download link as email using Flask API?

I need code to create a one-time download link for file uploaded using Flask. This link should be sent as email to the client. I have been able to create the dynamic link as per this solution:Link generator using django or any python module Modified part of the code(for flask):

def genUrl(filepath, fname):   # create a onetime salt for randomness   salt = ''.join(['{0}'.format(random.randrange(10) for i in range(10))])   key = hashlib.md5('{0}{1}'.format(salt, filepath)).hexdigest()   s = select([msettings.c.DL_URL])   rs = conn.execute(s).fetchone()   newpath = os.path.join(rs[msettings.c.DL_URL], key)   shutil.copy2(filepath, newpath)   ins = my_dlink.insert().values(key=key,                                  download_date=datetime.datetime.utcnow(),                                  orgpath=filepath,                                  newpath=newpath                                  )   rs1 = conn.execute(ins)   print rs1.inserted_primary_key[0], 'inserted_primary_key'   rs1.url = "{0}/{1}/{2}".format(       rs[msettings.c.DL_URL], key, os.path.basename(fname))    return rs1.url  @app.route('/archival/api/v1.0/archival_docs/<int:arc_file_id>/url',            methods=['POST']) def generate_one_time_download_url_for_file(arc_file_id):     path = ''     s = select([archival_docs]).where(archival_docs.c.id == arc_file_id)     rs = conn.execute(s).fetchone()     if rs:         path = os.path.join(("%s/%s" %                              (rs[archival_docs.c.path_map],                               rs[archival_docs.c.stored_name].encode('utf-8'))))     new_link = genUrl(path, rs[archival_docs.c.stored_name])      # Use BytesIO instead of StringIO here.     buffer = BytesIO()     buffer.seek(0)     content_type = mimetypes.guess_type(path)[0]     print content_type, 'content_type'     return send_file(buffer, as_attachment=True,                      attachment_filename=rs[archival_docs.c.stored_name],                      mimetype='text/plain')#content_type)      # response = make_response(send_file(path))     # response.headers["Content-Disposition"] = \     #     "attachment; " \     #     "filename={ascii_filename};" \     #     "filename*=UTF-8''{utf_filename}".format(     #     ascii_filename=rs[archival_docs.c.stored_name],     #     utf_filename=(os.path.basename(path))     #     )     # print response     #return response 

How to send this as 1 time download link to client? After client downloads, this link should get disabled.i.e response should indicate that the file was downloaded(cron job should take care of it). What should be the exact code changes?

1 Answers

Answers 1

There is some information missing wrt performance, security etc and I am also unclear on what the cron job "should take care of", but from what I do understand, I have the impression you may be going about it the wrong way.

What I understand is that you want a "file" model with a status available (i.e. it was uploaded) or unavailable (i.e. it was already downloaded) and the following "pages":

• upload new file
• download file, with the following responses:
  • a 404 if the file does not exist
  • the file download if it was not downloaded yet
  • a 'too late' message if it was already downloaded

Intermezzo on performance and security:

Typically, you want to bypass Flask to download static content and make the web server deal with this directly, but if you need to control file access + messages shown based on availability and download status of the file, it is easier to keep the same route and let Flask reply with the appropriate response.

Moreover, Flask has the send_file and send_from_directory functionality which does already quite some performance optimisation for you.

Nevertheless, it is possible to keep the file name + location + status seperate from the actual file and do a redirect to a static file download instead of using the Flask send_file functionality.

the "upload new file" function will for instance:

• accept the file
• create a new GUID
• store the file in a dedicated folder (include the GUID in the name, so two files with the same name don't overwrite each other)
• set initial state ("downloadable")
• send an email with a download link containing the GUID
• optional: store the file info in the db

the "download file" function will

• check the GUID parameter from the link
• check if the file exists / has existed
• "never heard of it" > 404 + send admin alert + ...
• "downloadable" > send file + change status + delete file + ...
• "already downloaded" > "too late" template + optionally increase counter + ...

I mention that the database is optional, because you could keep the downloadable files in one folder and move them to a "downloaded" folder when that happens and you can deduce the status of the different files from reading the folder contents, but I don't know what your constraints are or what else you may want to monitor.

Hope this helps you...

Read More

Avoid DB Session addition marshmallow object while using marshmallow sql-alchemy object

Is there a way to avoid inserting the data into session while using Marshmallow - sqlalchemy

sqlalchemy marshmallow avoid loading into session

Ref: https://marshmallow-sqlalchemy.readthedocs.org/en/latest/

Because we tried to manage the objects by ourself. Will add into the session if required , but for validation I need to use load ()

author = Author(name='Chuck Paluhniuk') book = Book(title='Fight Club', author=author) session.add(author) session.add(book) session.commit()  author_schema.dump(author).data # {'books': [123], 'id': 321, 'name': 'Chuck Paluhniuk'}  author_schema.load(dump_data, session=session).data # <Author(name='Chuck Paluhniuk')> 

The work around to avoid this issue , after I tried with load I can call DB Session.close() to ignore the temporary data. But again I need to get the session to flush the data into DB .

Please advice. Thanks for your help

2 Answers

Answers 1

Adding the object to the session has no bad effect on your code. As you said you're gonna do it yourself later. So there is no differences between adding it manually or by load method.

author = author_schema.load(dump_data, session=session) # now author has been added to session by load method # do whatever you want db.session.commit() # This is enough to write changes you've made # or if you don't want to save anything just db.session.rollback() 

I really didn't get why you want to prevent adding the object to the session by load's method

Answers 2

The call to the schema's load() doesn't automatically add the object to the session:

>>> from models import session >>> from schema import author_schema >>> a = {'id': 1, 'name': 'Chuck Paluhniuk'} >>> obj = author_schema.load(a, session=session) >>> session.new IdentitySet([]) >>> session.add(obj.data) >>> session.new IdentitySet([<Author(name=u'Chuck Paluhniuk')>]) >>> session.commit() >>> session.new IdentitySet([]) >>> session.identity_map.items() [((<class 'models.Author'>, (1,)), <Author(name=u'Chuck Paluhniuk')>)] 

The object must be making it into the session another way. Perhaps you could post some of your code exhibiting the issue?

Read More