Django: Sessions not working as expected on Heroku

Users keep getting logged out and sessions are not persisting on my Django app on Heroku. Users can log in, but they will be randomly logged out—even on the /admin/ site.

Is there anything I'm doing wrong with my Django/Heroku config?

Currently running Django 1.11.16 on Standard Dynos.

settings.py

SECRET_KEY = os.environ.get("SECRET_KEY", "".join(random.choice(string.printable) for i in range(40)))  SESSION_COOKIE_DOMAIN = ".appname.com" CSRF_COOKIE_DOMAIN = ".appname.com"  SECURE_SSL_REDIRECT = True  # ...  MIDDLEWARE_CLASSES = [     'django.middleware.security.SecurityMiddleware',     'django.contrib.sessions.middleware.SessionMiddleware',     'django.middleware.common.CommonMiddleware',     'django.middleware.csrf.CsrfViewMiddleware',     'django.contrib.auth.middleware.AuthenticationMiddleware',     'django.contrib.auth.middleware.SessionAuthenticationMiddleware',     'django.contrib.messages.middleware.MessageMiddleware',     'django.middleware.clickjacking.XFrameOptionsMiddleware', ]   TEMPLATES = [     {         'BACKEND': 'django.template.backends.django.DjangoTemplates',         'DIRS': [os.path.join(BASE_DIR, 'templates/')],         'APP_DIRS': True,         'OPTIONS': {             'context_processors': [                 'django.template.context_processors.debug',                 'django.template.context_processors.request',                 'django.template.context_processors.csrf',                 'django.contrib.auth.context_processors.auth',                 'django.contrib.messages.context_processors.messages',             ],         },     }, ]  # ...  DATABASES = {     'default': {         'ENGINE': 'django.db.backends.postgresql_psycopg2',         'NAME': 'appname',     } }  # https://devcenter.heroku.com/articles/python-concurrency-and-database-connections db_from_env = dj_database_url.config(conn_max_age=500) DATABASES['default'].update(db_from_env) 

1 Answers

Answers 1

The problem was that SECRET_KEY was not static on Heroku. The SECRET_KEY changing was breaking sessions. The fix is to add a static SECRET_KEY to Heroku config:

heroku config:set SECRET_KEY=`openssl rand -base64 32` 

Read More

Constant Validation Accuracy with a high loss in machine learning

I'm currently trying to do create an image classification model using Inception V3 with 2 classes. I have 1428 images which are balanced about 70/30. When I run my model I get a pretty high loss of as well as a constant validation accuracy. What might be causing this constant value?

data = np.array(data, dtype="float")/255.0 labels = np.array(labels,dtype ="uint8")  (trainX, testX, trainY, testY) = train_test_split(                             data,labels,                              test_size=0.2,                              random_state=42)   img_width, img_height = 320, 320 #InceptionV3 size  train_samples =  1145  validation_samples = 287 epochs = 20  batch_size = 32  base_model = keras.applications.InceptionV3(         weights ='imagenet',         include_top=False,          input_shape = (img_width,img_height,3))  model_top = keras.models.Sequential() model_top.add(keras.layers.GlobalAveragePooling2D(input_shape=base_model.output_shape[1:], data_format=None)), model_top.add(keras.layers.Dense(350,activation='relu')) model_top.add(keras.layers.Dropout(0.2)) model_top.add(keras.layers.Dense(1,activation = 'sigmoid')) model = keras.models.Model(inputs = base_model.input, outputs = model_top(base_model.output))   for layer in model.layers[:30]:   layer.trainable = False  model.compile(optimizer = keras.optimizers.Adam(                     lr=0.00001,                     beta_1=0.9,                     beta_2=0.999,                     epsilon=1e-08),                     loss='binary_crossentropy',                     metrics=['accuracy'])  #Image Processing and Augmentation  train_datagen = keras.preprocessing.image.ImageDataGenerator(           zoom_range = 0.05,           #width_shift_range = 0.05,            height_shift_range = 0.05,           horizontal_flip = True,           vertical_flip = True,           fill_mode ='nearest')   val_datagen = keras.preprocessing.image.ImageDataGenerator()   train_generator = train_datagen.flow(         trainX,          trainY,         batch_size=batch_size,         shuffle=True)  validation_generator = val_datagen.flow(                 testX,                 testY,                 batch_size=batch_size)  history = model.fit_generator(     train_generator,      steps_per_epoch = train_samples//batch_size,     epochs = epochs,      validation_data = validation_generator,      validation_steps = validation_samples//batch_size,     callbacks = [ModelCheckpoint]) 

This is my log when I run my model:

Epoch 1/20 35/35 [==============================]35/35[==============================] - 52s 1s/step - loss: 0.6347 - acc: 0.6830 - val_loss: 0.6237 - val_acc: 0.6875  Epoch 2/20 35/35 [==============================]35/35 [==============================] - 14s 411ms/step - loss: 0.6364 - acc: 0.6756 - val_loss: 0.6265 - val_acc: 0.6875  Epoch 3/20 35/35 [==============================]35/35 [==============================] - 14s 411ms/step - loss: 0.6420 - acc: 0.6743 - val_loss: 0.6254 - val_acc: 0.6875  Epoch 4/20 35/35 [==============================]35/35 [==============================] - 14s 414ms/step - loss: 0.6365 - acc: 0.6851 - val_loss: 0.6289 - val_acc: 0.6875  Epoch 5/20 35/35 [==============================]35/35 [==============================] - 14s 411ms/step - loss: 0.6359 - acc: 0.6727 - val_loss: 0.6244 - val_acc: 0.6875  Epoch 6/20 35/35 [==============================]35/35 [==============================] - 15s 415ms/step - loss: 0.6342 - acc: 0.6862 - val_loss: 0.6243 - val_acc: 0.6875 

2 Answers

Answers 1

I think you have too low learning rate and too few epochs. try with lr = 0.001 and epochs = 100.

Answers 2

Your accuracy is 68.25%. Given that your classes are split roughly 70/30 it is likely that your model is just predicting the same thing every time, ignoring the input. That would give the accuracy you are seeing. Your model has not yet learned from your data.

As Novak said, your learning rate seems very low, so maybe try increasing that first to see if that helps.

Read More

User Interface for filtering objects in Python

In my application I have a Job class as defined/outlined below. Instance of this job class represents a particular Job run. Job can have multiple checkpoints and each checkpoint can have multiple commands.

Job  - JobName  - [JobCheckpoint]  - StartTime  - EndTime  - Status  - ...  JobCheckpoint  - JobCheckpointName  - [JobCommand]  - StartTime  - EndTime  - Status  - ...  JobCommand  - JobCommandName  - [Command]  - StartTime  - EndTime  - Status   - ... 

At any given day there are like 100k different jobs that runs. I want to design a user interface in Python for querying these job objects. For example users want to query

1. Jobs that ran between x and y interval.
2. Jobs that run command x
3. Jobs in failed state.
4. All checkpoints/commands of a particular job.
5. And many more...

To solve this, I was thinking of providing following methods in user interface.

1. List getJobs(Filter)
2. List getCommands(Job)
3. List getCheckpoints(Job)

I am not sure

1. How Filter class will look like?
2. Is returning List of domain objects correct or should I return list of dict?
3. Should I take dict as an input or defined classes as an input.
4. Whether this is a best design.

1 Answers

Answers 1

These are partially subjective questions. But I'll have a go at answering some of them to the best of my current knowledge and the information available in the question posed.

How Filter class will look like?

That could depend for instance on the storage mechanism. Is it stored in-memory as a bunch of Python objects or is it first taken out of an SQL database or perhaps a NoSQL database.

If it's taken from an SQL database you can take advantage of the filtering mechanism of SQL. It's after all a (Structured) Query Language.

In that case your Filter class would be like a translator of field values to a bunch of SQL operators/conditions.

If it's a bunch of Python objects without some database mechanism to use for querying your data then you might need to think of your own query/filter methods.

A Filter class might be using a Condition class and an Operator class. Maybe you have an Operator class as an abstract class and have 'glue' operators to glue conditions together (AND/OR). And another kind of operators to compare a property of a domain object with a value.

For the latter, even if you are not designing a 'filter language' for it, you could get some inspiration from an API querying format, specified here for Flask-Restless: https://flask-restless.readthedocs.io/en/stable/searchformat.html#query-format

Surely if you are designing a query interface to e.g. a REST API, Flask-Restless's query-format could give you some inspiration of how to tackle the querying.

Is returning List of domain objects correct or should I return list of dict?

Returning a list of domain objects has the advantage of being able to use inheritance. That's at least one possible advantage.

A rough sketch of certain classes:

from abc import ABCMeta, abstractmethod from typing import List  class DomainObjectOperatorGlue(metaclass=ABCMeta):         @abstractmethod     def operate(self, haystack: List['DomainObject'], criteria:          List['DomainObject']) -> List['DomainObject']:         pass  class DomainObjectFieldGlueOperator(metaclass=ABCMeta):     @abstractmethod     def operate(self, conditions: List[bool]) -> bool:         pass  class DomainObjectFieldGlueOperatorAnd(DomainObjectFieldGlueOperator):     def operate(self, conditions: List[bool]) -> bool:         # If all conditions are True then return True here,         # otherwise return False.         # (...)         pass  class DomainObjectFieldGlueOperatorOr(DomainObjectFieldGlueOperator):     def operate(self, conditions: List[bool]) -> bool:         # If only one (or more) of the conditions are True then return True         # otherwise, if none are True, return False.         # (...)         pass   class DomainObjectOperatorAnd(DomainObjectOperatorGlue):     def __init__(self):         pass      def operate(self, haystack: 'JobsCollection', criteria:  List['DomainObject']) -> List['DomainObject']:         """         Returns list of haystackelements or empty list.         Includes haystackelement if all (search) 'criteria' elements  (DomainObjects) are met for haystackelement (DomainObject).         """         result = []         for haystackelement in haystack.jobs:             # AND operator wants all criteria to be True for haystackelement (Job)         # to be included in returned search results.         criteria_all_true_for_haystackelement = True         for criterium in criteria:             if haystackelement.excludes(criterium):                 criteria_all_true_for_haystackelement = False                 break         if criteria_all_true_for_haystackelement:             result.append(haystackelement)     return result  class DomainObjectOperatorOr(DomainObjectOperatorGlue):     def __init__(self):         pass  def operate(self, haystack: List['DomainObject'], criteria: List['DomainObject']) -> List['DomainObject']:     """     Returns list of haystackelements or empty list.     Includes haystackelement if all (search) 'criteria' elements (DomainObjects) are met for haystackelement (DomainObject).     """     result = []     for haystackelement in haystack:         # OR operator wants at least ONE criterium to be True for haystackelement         # to be included in returned search results.         at_least_one_criterium_true_for_haystackelement = False         for criterium in criteria:             if haystackelement.matches(criterium):                 at_least_one_criterium_true_for_haystackelement = True                 break         if at_least_one_criterium_true_for_haystackelement:             result.append(haystackelement)     return result  class DomainObjectFilter(metaclass=ABCMeta):     def __init__(self, criteria: List['DomainObject'], criteria_glue:  DomainObjectOperatorGlue):         self.criteria = criteria         self.criteria_glue = criteria_glue      @abstractmethod     def apply(self, haystack: 'JobsCollection') -> List['DomainObject']:         """        Applies filter to given 'haystack' (list of jobs with sub-objects in there);     returns filtered list of DomainObjects or empty list if none found     according to criteria (and criteria glue).         """         return self.criteria_glue.operate(haystack, self.criteria)  class DomainObject(metaclass=ABCMeta):     def __init__(self):         pass      @abstractmethod     def matches(self, domain_object: 'DomainObject') -> bool:         """ Returns True if this DomainObject matches specified DomainObject,     False otherwise.      """     pass  def excludes(self, domain_object: 'DomainObject') -> bool:     """     Convenience method; the inverse of includes-method.     """     return not self.matches(domain_object)   class Job(DomainObject):     def __init__(self, name, start, end, status, job_checkpoints:  List['JobCheckpoint']):         self.name = name         self.start = start         self.end = end         self.status = status         self.job_checkpoints = job_checkpoints      def matches(self, domain_object: 'DomainObject', field_glue:  DomainObjectFieldGlueOperator) -> bool:         """         Returns True if this DomainObject includes specified DomainObject,      False otherwise.          """         if domain_object is Job:             # See if specified fields in search criteria (domain_object/Job) matches this job.             # Determine here which fields user did not leave empty,             # and guess for sensible search criteria.             # Return True if it's  a match, False otherwise.             condition_results = []             if domain_object.name != None:                 condition_results.append(domain_object.name in self.name)             if domain_object.start != None or domain_object.end != None:                 if domain_object.start == None:                     # ...Use broadest start time for criteria here...                     # time_range_condition = ...                     condition_results.append(time_range_condition)                                  elif domain_object.end == None:                     # ...Use broadest end time for criteria here...                     # time_range_condition = ...                     condition_results.append(time_range_condition)                                  else:                     # Both start and end time specified; use specified time range.                 # time_range_condition = ...                 condition_results.append(time_range_condition)             # Then evaluate condition_results;             # e.g. return True if all condition_results are True here,             # false otherwise depending on implementation of field_glue class:             return field_glue.operate(condition_results)     elif domain_object is JobCheckpoint:         # Determine here which fields user did not leave empty,         # and guess for sensible search criteria.         # Return True if it's  a match, False otherwise.         # First establish if parent of JobCheckpoint is 'self' (this job)         # if so, then check if search criteria for JobCheckpoint match,         # glue fields with something like:         return field_glue.operate(condition_results)     elif domain_object is JobCommand:         # (...)         if domain_object.parent_job == self:             # see if conditions pan out             return field_glue.operate(condition_results)  class JobCheckpoint(DomainObject):     def __init__(self, name, start, end, status, job_commands: List['JobCommand'], parent_job: Job):        self.name = name         self.start = start         self.end = end         self.status = status        self.job_commands = job_commands         # For easier reference;         # e.g. when search criteria matches this JobCheckpoint         # then Job associated to it can be found         # more easily.         self.parent_job = parent_job  class JobCommand(DomainObject):     def __init__(self, name, start, end, status, parent_checkpoint: JobCheckpoint, parent_job: Job):         self.name = name         self.start = start         self.end = end         self.status = status         # For easier reference;         # e.g. when search criteria matches this JobCommand         # then Job or JobCheckpoint associated to it can be found         # more easily.         self.parent_checkpoint = parent_checkpoint         self.parent_job = parent_job  class JobsCollection(DomainObject):     def __init__(self, jobs: List['Job']):          self.jobs = jobs      def get_jobs(self, filter: DomainObjectFilter) -> List[Job]:         return filter.apply(self)      def get_commands(self, job: Job) -> List[JobCommand]:         """         Returns all commands for specified job (search criteria).         """         result = []         for some_job in self.jobs:             if job.matches(some_job):                 for job_checkpoint in job.job_checkpoints:                     result.extend(job_checkpoint.job_commands)          return result      def get_checkpoints(self, job: Job) -> List[JobCheckpoint]:         """         Returns all checkpoints for specified job (search criteria).         """         result = []         for some_job in self.jobs:             if job.matches(some_job):                 result.extend(job.job_checkpoints)         return result 

Read More

Python flask saml throwing saml2.sigver.SigverError Error Message

Has anyone succesfully implemented flask-saml using Windows as dev environment, Python 3.6 and Flask 1.0.2?

I was given the link to the SAML METADATA XML file by our organisation and had it configured on my flask app.

app.config.update({     'SECRET_KEY': 'changethiskeylaterthisisoursecretkey',     'SAML_METADATA_URL': 'https://<url>/FederationMetadata.xml', })  flask_saml.FlaskSAML(app) 

According to the documentation this extension will setup the following routes:

• /saml/logout/: Log out from the application. This is where users go if they click on a “Logout” button.

• /saml/sso/: Log in through SAML.

• /saml/acs/: After /saml/sso/ has sent you to your IdP it sends you back to this path. Also your IdP might provide direct login without needing the /saml/sso/ route.

When I go to one of the routes http://localhost:5000/saml/sso/ I get the error below

saml2.sigver.SigverError saml2.sigver.SigverError: Cannot find ['xmlsec.exe', 'xmlsec1.exe']

I then went to this site https://github.com/mehcode/python-xmlsec/releases/tag/1.3.5 to get xmlsec and install it. However, I'm still getting the same issue.

Here is a screenshot of how I installed xmlsec

where does not seem to find the xmlsec.exe

1 Answers

Answers 1

documentationis asking to have xmlsec1 pre-installed. What you installed is a python binding to xmlsec1.

Get a windows build of xmlsec1 from here or build it from source And make it available in the PATH.

Read More

PyInstaller ImportError DLL not found when testing EXE on other computer

I built an EXE file from a Python script using PyInstaller, using

pyinstaller --onefile myscript.py 

Packages I used:

pandas, numpy, imutils, opencv, logging, os, random, json, string, csv, datetime, uuid 

The EXE runs fine on my PC. However, when I try it on another PC I get the error shown in this screenshot: https://www.screencast.com/t/msZrURL4v

Any idea what the problem is?

0 Answers

Read More

Get top 5 values where key total is less than or equal to X

Currently I have a list of items someone can buy as follows:

my_list = [     ('Candy', 1.0, 20.5),     ('Soda', 3.0, 10.25),     ('Coffee', 1.2, 20.335),     ('Soap', 1.2, 11.5),     ('Spoon', 0.2, 2.32),     ('Toast', 3.2, 12.335),     ('Toothpaste', 3, 20.5),     ('Creamer', .1, 5.5),     ('Sugar', 2.2, 5.2), ] 

Each item is set up like this:

('Item Name', ItemCost, ItemValue) 

I have the list pulling the items with the top 5 ItemValue.

print nlargest(5, my_list, key=itemgetter(2)) >>> [         ('Candy', 1.0, 20.5),         ('Toothpaste', 3, 20.5),         ('Coffee', 1.2, 20.335),         ('Toast', 3.2, 12.335),         ('Soap', 1.2, 11.5),     ] 

I am trying to retrieve a result where I get the top 5 total ItemValue where the top 5 total ItemCost is equal or less than 6.

Any suggestions?

9 Answers

Answers 1

You can filter first, and use all following nlargest on your filtered list.

f = [(a,b,c) for (a,b,c) in my_list if b <= 6] 

But for data manipulation like this, pandas can be very useful. Take, for example

df = pd.DataFrame(my_list, columns=('ItemName', 'ItemCost', 'ItemValue'))      ItemName    ItemCost    ItemValue 0   Candy       1.0         20.500 1   Soda        3.0         10.250 2   Coffee      1.2         20.335 3   Soap        1.2         11.500 4   Spoon       0.2         2.320 5   Toast       3.2         12.335 6   Toothpaste  3.0         20.500 7   Creamer     0.1         5.500 8   Sugar       2.2         5.200  >>> df[df.ItemCost <= 6]      ItemName    ItemCost    ItemValue 0   Candy       1.0         20.500 1   Soda        3.0         10.250 2   Coffee      1.2         20.335 3   Soap        1.2         11.500 4   Spoon       0.2         2.320 5   Toast       3.2         12.335 6   Toothpaste  3.0         20.500 7   Creamer     0.1         5.500 8   Sugar       2.2         5.200  >>> df[df.ItemCost <= 6].nlargest(n=5, columns=['ItemValue'])       ItemName    ItemCost    ItemValue 0   Candy       1.0         20.500 6   Toothpaste  3.0         20.500 2   Coffee      1.2         20.335 5   Toast       3.2         12.335 3   Soap        1.2         11.500 

---

If you want, you can first get the nsmallest of the ItemCost and just then get the nlargest

df.nsmallest(n=5, columns=['ItemCost']).nlargest(n=5, columns=['ItemValue'])          ItemName    ItemCost    ItemValue 0   Candy       1.0         20.500 2   Coffee      1.2         20.335 3   Soap        1.2         11.500 7   Creamer     0.1         5.500 4   Spoon       0.2         2.320 

Answers 2

Not sure if this is what you asking,

I would first create all possible combinations of 5 elements from my_list

itertools.combinations(my_list, 5) 

Then i would find all possible combinations in result where total item cost would be less than or equal to 6.

f = [element for element in itertools.combinations(my_list, 5) if  sum([e[1] for e in element]) <=6] 

Now, I would find that element where total itemValue is the greatest

h = [sum([g[2] for g in e]) for e in f] 

The index of element with maximum itemValue is

index = h.index(max(h)) 

Now, you can find that element in f.

f[index] 

The answer i got is

 Candy        1.0  20.5  Coffee       1.2  20.335  Spoon        0.2  2.32  Toothpaste   3    20.5  Creamer      0.1  5.5 

Answers 3

First you want to filter the list by the ItemCost:

• Which can be done by: filtered_generator = filter(lambda x: x[1] <= 6, my_list)

• Or in a more python-like way filtered_list = [x for x in my_list if x[1] <=6]

• And to keep it a generator to save memory just use parentheses instead of the square brackets.

Then you want to get the n largest items:

• You can use heapq.nlargest: nlargest(5, filtered_iter, key=lambda x:x[2])
• or implement similar function yourself.

filtered_iter can be the list or one of the generators.

Answers 4

from operator import itemgetter from itertools import combinations from beautifultable import BeautifulTable  def pretty_print( lst):     table = BeautifulTable()     table.column_headers = ['Item Name','ItemCost','ItemValue']     if lst:         for item_specs in lst:             table.append_row(item_specs)         print(table)   def get_total_cost( lst):     return sum(item_specs[1] for item_specs in lst)  def get_total_Value( lst):     return sum(item_specs[2] for item_specs in lst)    def best_comb( item_list, number_of_items_to_pick, cost_constraint):       k = number_of_items_to_pick      item_list.sort(key=itemgetter(2), reverse=True) # sorting list by ItemValue      k_top_value_item_lst = item_list[:5] # picking top k items from list      total_cost = get_total_cost(k_top_value_item_lst)       def generateCombinations( take_default_val_for_best_result = True):         k_len_combination_list = list(combinations( item_list, k))          if take_default_val_for_best_result:             best_result = []# which meets total itemCost <= 6 condition and have highest total of ItemValue              best_result_sum = [0,0] # ItemCost, ItemValue             else:             best_result = k_top_value_item_lst             best_result_sum = [total_cost, get_total_Value(best_result)]           best_alternative_lst = [] # if there are any other combination which offer same Value for Cost          # ignore first comb as its been already suggested to user         for comb in k_len_combination_list:              temp_sum = [None,None]             temp_sum[0] = get_total_cost( comb)             reset_best = False              if  temp_sum[0] <= cost_constraint:                 temp_sum[1] = get_total_Value( comb)                  if best_result_sum[1] < temp_sum[1]:                     reset_best = True                  elif best_result_sum[1] == temp_sum[1]:                     if temp_sum[0] < best_result_sum[0]:                         reset_best = True                     elif temp_sum[0] == best_result_sum[0]:                         # since ItemValue as well as ItemCost are equivalent to best_result this comb is great alternative                         if comb != tuple(best_result):                             best_alternative_lst.append(comb)                  if reset_best:                     best_result = comb                     best_result_sum[1] = temp_sum[1]                     best_result_sum[0] = temp_sum[0]          print('Best Combination:')         if best_result:             pretty_print(best_result)         else:             print('not found')          if gen_alternative:             print('\nBest Alternative Combination:')             if best_alternative_lst:                 for idx,alter_comb in enumerate( best_alternative_lst):                     comb_id = idx+1                     print('combination_id ',comb_id)                     pretty_print(alter_comb)             else:                 print('not found')       if total_cost > cost_constraint:         generateCombinations()      else:         if gen_alternative:             generateCombinations(take_default_val_for_best_result = False)          else:             print('Best Combination:')             pretty_print(k_top_value_item_lst)    my_list = [     ('Candy', 2.0, 20.5),     ('Soda', 1.5, 25.7 ),      ('Coffee', 2.4, 25.7 ),     ('Soap', 1.2,20),     ('Spoon',1.2,20 ),      ('Toast',1.2,22 ),     ('Toothpaste',0.8, 20 ),      ('Creamer',0.8, 22),     ('Sugar',2.0, 20.5 ), ]  gen_alternative = input('do you want to generate alternative combinations: y/n ')[0].lower() == 'y'  best_comb( my_list, 5, 6) 

Answer to modified list ( to show extra feature)

do you want to generate alternative combinations: y/n Y Best Combination: +------------+----------+-----------+ | Item Name  | ItemCost | ItemValue | +------------+----------+-----------+ |    Soda    |   1.5    |   25.7    | +------------+----------+-----------+ |   Toast    |   1.2    |    22     | +------------+----------+-----------+ |  Creamer   |   0.8    |    22     | +------------+----------+-----------+ |    Soap    |   1.2    |    20     | +------------+----------+-----------+ | Toothpaste |   0.8    |    20     | +------------+----------+-----------+  Best Alternative Combination: combination_id  1 +------------+----------+-----------+ | Item Name  | ItemCost | ItemValue | +------------+----------+-----------+ |    Soda    |   1.5    |   25.7    | +------------+----------+-----------+ |   Toast    |   1.2    |    22     | +------------+----------+-----------+ |  Creamer   |   0.8    |    22     | +------------+----------+-----------+ |   Spoon    |   1.2    |    20     | <--- +------------+----------+-----------+ | Toothpaste |   0.8    |    20     | +------------+----------+-----------+       

Answer to your original list

Best Combination: +------------+----------+-----------+ | Item Name  | ItemCost | ItemValue | +------------+----------+-----------+ |   Candy    |   1.0    |   20.5    | +------------+----------+-----------+ | Toothpaste |    3     |   20.5    | +------------+----------+-----------+ |   Coffee   |   1.2    |  20.335   | +------------+----------+-----------+ |  Creamer   |   0.1    |    5.5    | +------------+----------+-----------+ |   Spoon    |   0.2    |   2.32    | +------------+----------+-----------+  Best Alternative Combination: not found 

Answers 5

Not totally sure of what you are trying to do, but...
If you are trying to retrieve the top x (5 or 6) based on itemcost sorted by lowest cost, you can try this.

x=5 sorted(my_list, key=lambda s : s[2])[:x]  This outputs the following:  [('Spoon', 0.2, 2.32), ('Sugar', 2.2, 5.2), ('Creamer', 0.1, 5.5), ('Soda', 3.0, 10.25), ('Soap', 1.2, 11.5)] 

Answers 6

from itertools import combinations  ...  total_cost = lambda item: int(sum(c for _, c, _ in item) <= 6) * sum(v for _, _ , v in item) chosen = max(combinations(my_list, 5), key=total_cost) 

Max can receive a function to specify max criteria.

The total_cost function, has a int(sum(c for _, c, _ in item) <= 6) portion that is 1 if total cost of the combination is less or equal then 6, and its 0 otherwise.

We then multiply this portion with the total sum of values.

combinations(my_list, 5) retrieves all possible combinations of my_list items having 5 items.

Printing chosen elements you have:

('Candy', 1.0, 20.5) ('Coffee', 1.2, 20.335) ('Spoon', 0.2, 2.32) ('Toothpaste', 3, 20.5) ('Creamer', 0.1, 5.5) 

Answers 7

Filter the list first:

print nlargest(5, [item for item in my_list if item[1]<=6], key=itemgetter(2))

You can do it with sorted too:

sorted([item for item in my_list if item[1]<=6], key=lambda x: x[1], reverse=True)[:5]

The above filters out items with ItemCost greater than 6, sorts your list descending based on ItemCost, and then returns the first 5 element

Answers 8

First using combinations we can get all possible combinations of 5 from my_list. From here we can use filter and only return combinations whose total ItemCost is less than or equal to 6. Finally we sort by the which group has the highest total ItemValue and we take the greatest one being l2[-1] we could use reverse = True and then it would be l2[0]

from itertools import combinations

l = list(combinations(my_list, 5)) l1 = list(filter(lambda x: sum([i[1] for i in x]) < 6, l)) l2 = sorted(l1, key=lambda x: sum([i[2] for i in x])) print(l2[-1]) 

(('Candy', 1.0, 20.5), ('Coffee', 1.2, 20.335), ('Spoon', 0.2, 2.32), ('Toothpaste', 3, 20.5), ('Creamer', 0.1, 5.5)) 

Answers 9

from itertools import combinations from functools import reduce  def get_valid_combs(lis):     "find all combinations that cost less than or equal to 6"      for i in combinations(lis, 5):         if reduce(lambda acc, x: acc + x[1], list(i), 0) <= 6:             yield list(i)  my_list = [     ('Candy', 1.0, 20.5),     ('Soda', 3.0, 10.25),     ('Coffee', 1.2, 20.335),     ('Soap', 1.2, 11.5),     ('Spoon', 0.2, 2.32),     ('Toast', 3.2, 12.335),     ('Toothpaste', 3, 20.5),     ('Creamer', .1, 5.5),     ('Sugar', 2.2, 5.2), ]  # find all valid combinations which  cost less than 6 valid_combinations = [i for i in get_valid_combs(my_list)]  #top_combinations_sorted = sorted(valid_combinations, key=lambda y: reduce(lambda acc, x: acc + x[2], [0]+y))  # of the valid combinations get the combination with highest total value best_combination = max(valid_combinations, key=lambda y: reduce(lambda acc, x: acc + x[2], y, 0))  print(best_combination) 

output:

[('Candy', 1.0, 20.5), ('Coffee', 1.2, 20.335), ('Spoon', 0.2, 2.32), ('Toothpaste', 3, 20.5), ('Creamer', 0.1, 5.5)] 

Read More

Using uWSGI to proxy certain requests

I am trying to send all requests /other to another server, say google for example. As far as I understand the config I should be able to do something like this in the config file:

[uwsgi] master = 1 buffer-size = 65535 die-on-term = true # HTTP http-socket = 0.0.0.0:80 # App module = manage:app # Async processes gevent = 100 processes = 4 route-if = equal:${PATH_INFO};/other http:216.58.204.78,www.google.com 

This does not work as the log just has

error routing request to http server 216.58.204.78 [pid: 9|app: -1|req: -1/11] 172.18.0.1 () {36 vars in 759 bytes} [Thu Sep 20 14:51:55 2018] GET /other => generated 0 bytes in 0 msecs via route() (HTTP/1.1 500) 0 headers in 0 bytes (1 switches on core 99) 

1 Answers

Answers 1

according to the uwsgi doc, you need to specify the external HTTP server address in form of HOST:PORT, for example, with config:

route-if = equal:${PATH_INFO};/s http:220.181.111.188:80,www.baidu.com 

result:

[pid: 5113|app: -1|req: -1/1] 127.0.0.1 () {24 vars in 249 bytes} [Tue Sep 25 17:58:33 2018] GET /s => generated 118322 bytes in 34 msecs via route() (HTTP/1.1 200) 18 headers in 944 bytes (0 switches on core 0) 

Read More

Temporary ftp server for testing

I want to write a test for my code which uses a ftp library and does upload data via ftp.

I would like to avoid the need for a real ftp server in my test.

What is the most simple way to test my code?

There are several edge-cases which I would like to test.

For example: my code tries to create a directory which already exists.

I want to catch the exception and do appropriate error handling.

I know that I could use the mocking library. I used it before. But maybe there is a better solution for this use case?

Update Why I don't want to do mocking: I know that I could use mocking to solve this. I could mock the library I use (I use ftputil from Stefan Schwarzer) and test my code this way. But what happens if I change my code and use a different ftp library in the future? Then I would need to re-write my testing code, too. I am lazy. I want to be able to rewrite the real code I am testing without touching the test code. But maybe I am still missing a cool way to use mocking.

2 Answers

Answers 1

Firstly to hey this or of the way. You aren't asking about Mocking, your question is about Faking.

• Fake, an implementation of an interface, which expresses correct behaviour, but cannot be used in production.

• Mock, an implementation of an interface that responds to interactions based on a scripted (script as in movie script, not uncompiled code) response.

• Stub, an implementation of an interface lacking any real implementation. Usually used in mcguffin style tests.

Notice that in every case the word "interface" is used.

Your question asks how to Fake a TCP port such that the behaviour is a FTP server, with STATE of a rw filesystem underneath.

This is hard.

It is much easier to MOCK an internal interface that throws when you call the mkdir function.

If you must FAKE a FTP server. I suggest creating a docker container with the server in the state you want and use docker to handle the repeatability and lifecycle of the FTP server.

Answers 2

ContextManager:

class FTPServerContext(object):      banner = 'FTPServerContext ready'      def __init__(self, directory_to_serve):         self.directory_to_serve = directory_to_serve      def __enter__(self):         cmd = ['serve_directory_via_ftp']         self.pipe = subprocess.Popen(cmd, cwd=self.directory_to_serve)         time.sleep(2) # TODO check banner via https://stackoverflow.com/a/4896288/633961      def __exit__(self, *args):         self.pipe.kill() 

console_script:

def serve_directory_via_ftp():     # https://pyftpdlib.readthedocs.io/en/latest/tutorial.html     authorizer = DummyAuthorizer()     authorizer.add_user('testuser-ftp', 'testuser-ftp-pwd', '.', perm='elradfmwMT')     handler = FTPHandler     handler.authorizer = authorizer     handler.banner = testutils.FTPServerContext.banner     address = ('localhost', 2121)     server = FTPServer(address, handler)     server.serve_forever() 

Usage in test:

def test_execute_job_and_create_log(self):     temp_dir = tempfile.mkdtemp()     with testutils.FTPServerContext(temp_dir) as ftp_context:         execute_job_and_create_log(...) 

Code is in the public domain under any license you want. It would great if you make this a pip installable package at pypi.org.

Read More

Python lagged series to Pyspark

I am trying to do adapt this Python code in pyspark:

from statsmodels.tsa.tsatools import lagmat  def lag_func(data,lag):     lag = lag     X = lagmat(data["diff"], lag)     lagged = data.copy()     for c in range(1,lag+1):         lagged["lag%d" % c] = X[:, c-1]     return lagged  def diff_creation(data):     data["diff"] = np.nan     data.ix[1:, "diff"] = (data.iloc[1:, 1].as_matrix() - data.iloc[:len(data)-1, 1].as_matrix())     return data 

The result is a dataframe with lagged columns.

I tried something like that:

class SerieMaker(Transformer):     def __init__(self, inputCol='f_qty_recalc', outputCol='serie', dateCol='dt_ticket_sale', idCol= ['id_store', 'id_sku'], serieSize=30):         self.inputCol = inputCol         self.outputCol = outputCol         self.dateCol = dateCol         self.serieSize = serieSize         self.idCol = idCol      def _transform(self, df):         window = Window.partitionBy(self.idCol).orderBy(self.dateCol)         series = []             df = df.withColumn('filled_serie', F.lit(0))          """ 30 days lag"""          for index in reversed(range(0, self.serieSize)):             window2 = Window.partitionBy(self.idCol).orderBy(self.dateCol).rowsBetween((self.serieSize - index), self.serieSize)             col_name = (self.outputCol + '%s' % index)             series.append(col_name)             df = df.withColumn(col_name, F.when(F.isnull(F.lag(F.col(self.inputCol), index).over(window)),                                                  F.first(F.col(self.inputCol),                                                          ignorenulls=True).over(window2)).otherwise(F.lag(F.col(self.inputCol),                                                                                                           index).over(window)))             df = df.withColumn('filled_serie', F.when(F.isnull(F.lag(F.col(self.inputCol), index).over(window)),                                                        (F.col('filled_serie') + 1)).otherwise(F.col('filled_serie')))             df = df.withColumn('rank', F.rank().over(window))             return df.withColumn(self.outputCol, F.col(*series)) 

My df looks like:

  id_sku|id_store|     dt_ticket_sale|f_qty_recalc|prc_sku|sales| +------------+--------+-------------------+------------+-------+-----+ |    514655.0|    1090|2017-12-20 00:00:00|           1|   1.23| 1.23| |    823259.0|     384|2017-12-20 00:00:00|           1|   2.79| 2.79| 

My expected output is some lag of fqty_recalc and at the beginning idsku idstore and date (not shown there):

    diff    lag1    lag2    lag3    lag4    lag5    lag6    lag7    lag8    lag9    ... lag20   lag21   lag22   lag23   lag24   lag25   lag26   lag27   lag28   lag29 0   NaN 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1   0.0 NaN 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 

0 Answers

Read More

Long running script from flask endpoint

I've been pulling my hair out trying to figure this one out, hoping someone else has already encountered this and knows how to solve it :)

I'm trying to build a very simple Flask endpoint that just needs to call a long running, blocking php script (think while true {...}). I've tried a few different methods to async launch the script, but the problem is my browser never actually receives the response back, even though the code for generating the response after running the script is executed.

I've tried using both multiprocessing and threading, neither seem to work:

# multiprocessing attempt @app.route('/endpoint') def endpoint():   def worker():     subprocess.Popen('nohup php script.php &', shell=True, preexec_fn=os.setpgrp)    p = multiprocessing.Process(target=worker)   print '111111'   p.start()   print '222222'   return json.dumps({     'success': True   })  # threading attempt @app.route('/endpoint') def endpoint():   def thread_func():     subprocess.Popen('nohup php script.php &', shell=True, preexec_fn=os.setpgrp)    t = threading.Thread(target=thread_func)   print '111111'   t.start()   print '222222'   return json.dumps({     'success': True   }) 

In both scenarios I see the 111111 and 222222, yet my browser still hangs on the response from the endpoint. I've tried p.daemon = True for the process, as well as p.terminate() but no luck. I had hoped launching a script with nohup in a different shell and separate processs/thread would just work, but somehow Flask or uWSGI is impacted by it.

Update

Since this does work locally on my Mac when I start my Flask app directly with python app.py and hit it directly without going through my Nginx proxy and uWSGI, I'm starting to believe it may not be the code itself that is having issues. And because my Nginx just forwards the request to uWSGI, I believe it may possibly be something there that's causing it.

Here is my ini configuration for the domain for uWSGI, which I'm running in emperor mode:

[uwsgi] protocol = uwsgi max-requests = 5000 chmod-socket = 660 master = True vacuum = True enable-threads = True auto-procname = True procname-prefix = michael- chdir = /srv/www/mysite.com module = app callable = app socket = /tmp/mysite.com.sock 

3 Answers

Answers 1

This kind of stuff is the actual and probably main use case for Python Celery (http://www.celeryproject.org/). As a general rule, do not run long running jobs that are CPU-bound in the wsgi process. It's tricky, it's inefficient, and most important thing, it's more complicated than setting up an async task in a celery worker. If you want to just prototype you can set the broker to memory and not using an external server, or run a single threaded redis on the very same machine.

This way you can launch the task, call task.result() which is blocking, but it blocks in an IO-bound fashion, or even better you can just return immediately by retrieving the task_id and build a second endpoint /result?task_id=<task_id> that checks if result is available:

result = AsyncResult(task_id, app=app) if result.state == "SUCCESS":    return result.get() else:    return result.state  # or do something else depending on the state 

This way you have a non-blocking wsgi app that does what is best suited for: short time CPU-unbound calls that have IO calls at most with OS-level scheduling, then you can rely directly to the wsgi server workers|processes|threads or whatever you need to scale the API in whatever wsgi-server like uwsgi, gunicorn, etc. for the 99% of workloads as celery scales horizontally by increasing the number of worker processes.

Answers 2

This approach works for me, it calls the timeout command (sleep 10s) in the command line and lets it work in the background. It returns the response immediately.

@app.route('/endpoint1') def endpoint1():     subprocess.Popen('timeout 10', shell=True)     return 'success1' 

However, not testing on WSGI server, but just locally.

Answers 3

Since this works locally, but doesn't with NGINX and uWSGI ...

• Have you tried adding processes = 2 and threads = 2 to your uWSGI config?
• Have you tried running this without NGINX?

Read More

creating and appending to a list in SQLAlchemy database table

I am learning SQLAlchemy and I am stuck. I have a SQL table (table1) has two fields: 'name' and 'other_names'

I have an excel file with two columns:

first_name alias    paul   patrick john   joe simon  simone john   joey john   jo 

I want to read the excel file into my table1, so that it looks like this (i.e. all of the aliases for the same line are on one row):

paul    patrick john    joe,joey,jo simon   simone 

This is the idea that I was trying to do. The code (with comments) that I tried:

for line in open('file.txt', 'r'): #for each line in the excel file         line = line.strip().split('\t') #split each line with a name and alias         first_name = line[0] #first name is the name before the tab         alias = line[1] #alias is the name after the tab         instance =          Session.query(session,tbs['table1'].name).filter_by(name=first_name) #look through the database table, by name field, and see if the first name is there          list_instance = [x[0] for x in instance] #make a list of first names already in database table         if first_name not in list_instance: #if the excel first name is not in the database table               alias_list = [] #make an empty list               alias_list.append(alias) #append the alias               name_obj = lib.get_or_create( #small function to make db object               session,               tbs["table1"],               name = first_name, #add first name to the name field               other_names = alias_list # add alias list to the other_names field             )          elif first_name in list_instance: #elif first name already in db              alias_list.append(alias) #append the alias to the alias list made above              name_obj = lib.get_or_create(              session,              tbs["table1"],              name = first_name,              other_names = alias_list #create object as before, but use updated alias list     ) 

The problem is that I can get the above code to run with no errors, but also the output is not an appended list, it is simply a database table that looks like the excel file; i.e.

name   alias paul   patrick john   joe simon  simone john   joey john   jo 

Could someone point out where I am going wrong, specifically, how do i amend this code? Please let me know if the question is unclear, I've tried to make it a simple example. Specifically, how do I initialise and add to lists as a field entry in a SQLalchemy db table.

Update 1: I have updated my code according to kind suggestion below. However I still have the issue. This is the full aim, code and test file: The aim:

I have a table in the database (see below for test file going into table).The table has two fields, name (the latin name e.g. homo sapiens) and other names (the common names e.g. human, man). I want to update a field (other names) in the table, so instead of having:

Rana rugosa human    Rana rugosa man  Rana rugosa frog     Rana rugosa cow 

I have:

Rana rugosa human,man,frog,cow 

The test_data file looks like this:

origin_organism        common_name         tested_organism Rana rugosa            human                - Rana rugosa            man                  - Rana rugosa            frog                 homo sapiens Rana rugosa            cow                  Rana rugosa Rana rugosa            frog                 Rana rugosa Rana rugosa            frog                 - Rana rugosa            frog                 - Rana rugosa            frog                homo sapiens -                      -                   - -                      -                   homo sapiens -                      -                   - -                      -                   - -                      -                   - -                      -                   - streptococcus pneumoniae    -              - 

The code:

import sys  from sqlalchemy.orm  import *  from sqlalchemy  import *  from dbn.sqlalchemy_module  import lib  import pd  engine = lib.get_engine(user="user", psw="pwd", db="db", db_host="111.111.111.11") Base = lib.get_automapped_base(engine) session = Session(engine) tbs = lib.get_mapped_classes(Base) session.rollback() df = pd.read_excel('test_data.xlsx', sheet_name = 'test2')     for index, row in df.iterrows():       origin_latin_name = row['origin_organism'].strip().lower()     other_names_name = row['common_name'].strip().lower()     tested_species = row['tested_organism'].strip().lower()   if origin_latin_name not in [None, "None", "", "-"]:     instance = [x[0] for x in Session.query(session,tbs['species'].name).filter_by(name=origin_latin_name).all()]     if origin_latin_name not in instance:         origin_species = lib.get_or_create(             session,             tbs["species"],             name = origin_latin_name,             other_names = other_names_name         )      elif origin_latin_name in instance:         other_names_query = Session.query(session,tbs['species'].other_names).filter_by(name=origin_latin_name)         other_names_query_list = [x for x in other_names_query]         original_list2 = list(set([y for y in x[0].split(',') for x in other_names_query_list]))         if other_names_name not in original_list2:             original_list2.append(other_names_name)             new_list = ','.join(original_list2)             new_names = {'other_names':','.join(original_list2)}          origin_species = lib.get_or_create(             session,             tbs["species"],             name = origin_latin_name,             other_names = new_list         ) 

The part from the elif statement doesn't work. I've ran into two problems:

(1) The most recent error I got: NameError: name 'new_list' is not defined

(2) another error I got is that I have another table further on

map1 = lib.get_or_create(     session,     tbs["map1"],     age_id_id = age,     name_id_id = origin_species.id     ) 

...and it said that origin_species cannot be found, but I think this is linked to the elif statement, that somehow the origin_species object is not being updated properly.

If anyone could help I would appreciate it.

1 Answers

Answers 1

Simple mistake. You aren't giving it a list. I'm not sure why they end up in different rows, however, I would change the following because at the moment I don't see where you split the names into a list, all I see is you assigning a string onto a list using append.

alias_list = alias.split(',') 

Which could also be:

alias_list = line[1].split(',') 

Output:

alias_list:    ['Name1','Name2','Name3'] 

Currently your code outputs:

alias_list = ['Name1,Name2,Name3'] 

Which, whilst it is technically a list by data type, it is a worthless list for the way you want to use it. This is because alias_list[0] would return the entire string, as opposed to 'Name1'

WORD OF WARNING:

Your code is creating a list unnecessarily. You don't need a list in your database, you can easily achieve what you wabt by using the string that is evaluated when you read the excel file.

What you should do IMHO is to store the string of names as a whole string, then if you need to query the aliases of someone, then you can split the string on the other side, if that makes sense?

Read More

Keras: Accuracy Drops While Finetuning Inception

I am having trouble fine tuning an Inception model with Keras.

I have managed to use tutorials and documentation to generate a model of fully connected top layers that classifies my dataset into their proper categories with an accuracy over 99% using bottleneck features from Inception.

import numpy as np from keras.preprocessing.image import ImageDataGenerator from keras.models import Sequential from keras.layers import Dropout, Flatten, Dense from keras import applications   # dimensions of our images. img_width, img_height = 150, 150  #paths for saving weights and finding datasets top_model_weights_path = 'Inception_fc_model_v0.h5' train_data_dir = '../data/train2' validation_data_dir = '../data/train2'   #training related parameters? inclusive_images = 1424 nb_train_samples = 1424 nb_validation_samples = 1424 epochs = 50 batch_size = 16   def save_bottlebeck_features():     datagen = ImageDataGenerator(rescale=1. / 255)      # build bottleneck features     model = applications.inception_v3.InceptionV3(include_top=False, weights='imagenet', input_shape=(img_width,img_height,3))      generator = datagen.flow_from_directory(         train_data_dir,         target_size=(img_width, img_height),         batch_size=batch_size,         class_mode='categorical',         shuffle=False)      bottleneck_features_train = model.predict_generator(         generator, nb_train_samples // batch_size)      np.save('bottleneck_features_train', bottleneck_features_train)      generator = datagen.flow_from_directory(         validation_data_dir,         target_size=(img_width, img_height),         batch_size=batch_size,         class_mode='categorical',         shuffle=False)      bottleneck_features_validation = model.predict_generator(         generator, nb_validation_samples // batch_size)      np.save('bottleneck_features_validation', bottleneck_features_validation)  def train_top_model():     train_data = np.load('bottleneck_features_train.npy')     train_labels = np.array(range(inclusive_images))      validation_data = np.load('bottleneck_features_validation.npy')     validation_labels = np.array(range(inclusive_images))      print('base size ', train_data.shape[1:])      model = Sequential()     model.add(Flatten(input_shape=train_data.shape[1:]))     model.add(Dense(1000, activation='relu'))     model.add(Dense(inclusive_images, activation='softmax'))     model.compile(loss='sparse_categorical_crossentropy',              optimizer='Adam',              metrics=['accuracy'])      proceed = True      #model.load_weights(top_model_weights_path)      while proceed:         history = model.fit(train_data, train_labels,               epochs=epochs,               batch_size=batch_size)#,               #validation_data=(validation_data, validation_labels), verbose=1)         if history.history['acc'][-1] > .99:             proceed = False      model.save_weights(top_model_weights_path)   save_bottlebeck_features() train_top_model() 

Epoch 50/50 1424/1424 [==============================] - 17s 12ms/step - loss: 0.0398 - acc: 0.9909

I have also been able to stack this model on top of inception to create my full model and use that full model to successfully classify my training set.

from keras import Model from keras import optimizers from keras.callbacks import EarlyStopping  img_width, img_height = 150, 150  top_model_weights_path = 'Inception_fc_model_v0.h5' train_data_dir = '../data/train2' validation_data_dir = '../data/train2'   #how many inclusive examples do we have? inclusive_images = 1424 nb_train_samples = 1424 nb_validation_samples = 1424 epochs = 50 batch_size = 16  # build the complete network for evaluation base_model = applications.inception_v3.InceptionV3(weights='imagenet', include_top=False, input_shape=(img_width,img_height,3))  top_model = Sequential() top_model.add(Flatten(input_shape=base_model.output_shape[1:])) top_model.add(Dense(1000, activation='relu')) top_model.add(Dense(inclusive_images, activation='softmax'))  top_model.load_weights(top_model_weights_path)  #combine base and top model fullModel = Model(input= base_model.input, output= top_model(base_model.output))  #predict with the full training dataset results = fullModel.predict_generator(ImageDataGenerator(rescale=1. / 255).flow_from_directory(         train_data_dir,         target_size=(img_width, img_height),         batch_size=batch_size,         class_mode='categorical',         shuffle=False)) 

inspection of the results from processing on this full model match the accuracy of the bottleneck generated fully connected model.

import matplotlib.pyplot as plt import operator  #retrieve what the softmax based class assignments would be from results resultMaxClassIDs = [ max(enumerate(result), key=operator.itemgetter(1))[0] for result in results]  #resultMaxClassIDs should be equal to range(inclusive_images) so we subtract the two and plot the log of the absolute value  #looking for spikes that indicate the values aren't equal  plt.plot([np.log(np.abs(x)+10) for x in (np.array(resultMaxClassIDs) - np.array(range(inclusive_images)))]) 

Here is the problem: When I take this full model and attempt to train it, Accuracy drops to 0 even though validation remains above 99%.

model2 = fullModel  for layer in model2.layers[:-2]:     layer.trainable = False  # compile the model with a SGD/momentum optimizer # and a very slow learning rate. #model.compile(loss='binary_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),  metrics=['accuracy'])  model2.compile(loss='categorical_crossentropy',              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),               metrics=['accuracy'])  train_datagen = ImageDataGenerator(rescale=1. / 255)  test_datagen = ImageDataGenerator(rescale=1. / 255)  train_generator = train_datagen.flow_from_directory(     train_data_dir,     target_size=(img_height, img_width),     batch_size=batch_size,     class_mode='categorical')  validation_generator = test_datagen.flow_from_directory(     validation_data_dir,     target_size=(img_height, img_width),     batch_size=batch_size,     class_mode='categorical')  callback = [EarlyStopping(monitor='acc', min_delta=0, patience=3, verbose=0, mode='auto', baseline=None)] # fine-tune the model model2.fit_generator(     #train_generator,     validation_generator,     steps_per_epoch=nb_train_samples//batch_size,     validation_steps = nb_validation_samples//batch_size,     epochs=epochs,     validation_data=validation_generator) 

Epoch 1/50 89/89 [==============================] - 388s 4s/step - loss: 13.5787 - acc: 0.0000e+00 - val_loss: 0.0353 - val_acc: 0.9937

and it gets worse as things progress

Epoch 21/50 89/89 [==============================] - 372s 4s/step - loss: 7.3850 - acc: 0.0035 - val_loss: 0.5813 - val_acc: 0.8272

The only thing I could think of is that somehow the training labels are getting improperly assigned on this last train, but I've successfully done this with similar code using VGG16 before.

I have searched over the code trying to find a discrepancy to explain why a model making accurate predictions over 99% of the time drops its training accuracy while maintaining validation accuracy during fine tuning, but I can't figure it out. Any help would be appreciated.

Information about the code and environment:

Things that are going to stand out as weird, but are meant to be that way:

• There is only 1 image per class. This NN is intended to classify objects whose environmental and orientation conditions are controlled. Their is only one acceptable image for each class corresponding to the correct environmental and rotational situation.
• The test and validation set are the same. This NN is only ever designed to be used on the classes it is being trained on. The images it will process will be carbon copies of the class examples. It is my intent to overfit the model to these classes

I am using:

• Windows 10
• Python 3.5.6 under Anaconda client 1.6.14
• Keras 2.2.2
• Tensorflow 1.10.0 as the backend
• CUDA 9.0
• CuDNN 8.0

I have checked out:

1. Keras accuracy discrepancy in fine-tuned model
2. VGG16 Keras fine tuning: low accuracy
3. Keras: model accuracy drops after reaching 99 percent accuracy and loss 0.01
4. Keras inception v3 retraining and finetuning error
5. How to find which version of TensorFlow is installed in my system?

but they appear unrelated.

2 Answers

Answers 1

Note: Since your problem is a bit strange and difficult to debug without having your trained model and dataset, this answer is just a (best) guess after considering many things that may have could go wrong. Please provide your feedback and I will delete this answer if it does not work.

Since the inception_V3 contains BatchNormalization layers, maybe the problem is due to (somehow ambiguous or unexpected) behavior of this layer when you set trainable parameter to False (1, 2, 3, 4).

Now, let's see if this is the root of the problem: as suggested by @fchollet, set the learning phase when defining the model for fine-tuning:

from keras import backend as K  K.set_learning_phase(0)  base_model = applications.inception_v3.InceptionV3(weights='imagenet', include_top=False, input_shape=(img_width,img_height,3))  for layer in base_model.layers:     layer.trainable = False  K.set_learning_phase(1)  top_model = Sequential() top_model.add(Flatten(input_shape=base_model.output_shape[1:])) top_model.add(Dense(1000, activation='relu')) top_model.add(Dense(inclusive_images, activation='softmax'))  top_model.load_weights(top_model_weights_path)  #combine base and top model fullModel = Model(input= base_model.input, output= top_model(base_model.output))  fullModel.compile(loss='categorical_crossentropy',              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),               metrics=['accuracy'])   ##################################################################### # Here, define the generators and then fit the model same as before # ##################################################################### 

---

Side Note: This is not causing any problem in your case, but keep in mind that when you use top_model(base_model.output) the whole Sequential model (i.e. top_model) is stored as one layer of fullModel. You can verify this by either using fullModel.summary() or print(fullModel.layers[-1]). Hence when you used:

for layer in model2.layers[:-2]:     layer.trainable = False  

you are actually not freezing the last layer of base_model as well. However, since it is a Concatenate layer, and therefore does not have trainable parameters, no problem occurs and it would behave as you intended.

Answers 2

Like the previous reply, I'll try to share some thoughts to see whether it helps.

There are a couple of things that called my attention (and maybe are worth reviewing). Note: some of them should have given you issues with the separate models as well.

• Correct if I'm wrong, but it seems you used sparse_categorical_crossentropy for the first training while you used categorical_crossentropy for the second one. Is it correct? Because I believe they assume labels differently (sparse assumes integers and the other assumes one-hot).
• Have you tried to set the layers you added in the end as trainable = True? I know that you have already set the others to trainable = False, but maybe that's something worth checking too.
• It seems the data generator is not making use of the default preprocessing function used in Inception v3, which uses a per-mean channel.
• Have you tried any experiment using Functional instead of Sequential API?

I hope that helps.

Read More

Pip --user installs package to Default user directory on Windows 10

I have a problem on Windows 10 where both Python 2.6 and 2.7 are installed.

python -m pip install myPack --no-index --find-links=. --user 

When running this command with user AutoUser it installs myPack to Default user directory C:\Users\Default\Python\Python27\site-packages or C:\Users\Default\Appdata\Roaming\Python\site-packages instead C:\Users\Autouser\Appdata\Roaming\Python\site-packages

• Installation is automatic soon after windows logon, but I can see in logs that "query user" returns a row with AutoUser (before calling pip).
• Other OS don't have this problem.
• Reproduction is unstable on Windows 10: maybe 1 time of 100.
• Truth that python 2.6 is also installed on these machines, but I'm not sure it is meaningful: 2.6 goes later than 2.7 in Path system variable. Here they write it could be a problem, but pip doesn't confuse python versions, it confuses users' directories.

---

Path:

C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts\;C:\Python26\;C:\Python26\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\ProgramData\chocolatey\bin; 

Python version:

python --version Python 2.7.13 

Pip version:

python -m pip --version Pip version: pip 9.0.1 from C:\Python27\lib\site-packages (python 2.7)  

1 Answers

Answers 1

You can try setting the install target with the --target option like so:

pip install --target=C:\Users\Autouser\Appdata\Roaming\Python\site-packages package_name 

If that doesn't work, another option is to try using --install-option like this:

pip install --install-option="--prefix=$PREFIX_PATH" package_name 

Finally, if all else fails, here's one more way to do it:

PYTHONUSERBASE=/path/to/install/to pip install --user 

You can specify which python version to install the package for by using python2.x -m pip install ...

Hopefully one of these helps you! :)

Read More

Is there a non-math version of matplotlib.ticker.LogFormatterSciNotation?

I am trying to plot a graph with a logarithmic y-axis using pgf_with_latex, i.e. all text formatting is done by pdflatex. In my matplotlib rc Parameters I define a font to be used. Here comes my problem: The standard matplotlib.ticker.LogFormatterSciNotation formatter used math text and therefore a math font, which does not fit the rest of the fonts (sans-serif).

How can I format the y-axis labels using a formatter from matplotlib.ticker so that I get the labels formatted as powers of 10 with superscripted powers? To be more specific: How do I get these yticklabels formatted the same way but with the font from the xticklabels?

I already tried using different formatters provided by matplotlib.ticker, but none of them has the exponents written the way I want.

Here is an example of what I mean with a MWE below.

import matplotlib as mpl  mpl.use('pgf') pgf_with_latex = {         "pgf.texsystem": "pdflatex",         "font.family": "sans-serif",         "text.usetex": False,         "pgf.preamble": [             r"\usepackage[utf8x]{inputenc}",             r"\usepackage{tgheros}",  # TeX Gyre Heros sans serif             r"\usepackage[T1]{fontenc}"             ]         }  mpl.rcParams.update(pgf_with_latex) import matplotlib.pyplot as plt  fig = plt.figure(figsize=[3, 2]) ax = fig.add_subplot(111) ax.set_yscale("log") ax.minorticks_off() ax.set_xlabel("sans-serif font label") ax.set_ylabel("math font label") plt.gca().set_ylim([1, 10000]) plt.gcf().tight_layout()   plt.savefig('{}.pdf'.format("test")) 

Caution: A TeX distribution has to be installed on your system to run this. I used MikTex 2.9. Also Python 3.6.2 and matplotlib 2.1.2.

2 Answers

Answers 1

You could subclass LogFormatterExponent to format the ticks with "10\textsuperscript{x}" where x is the exponent. This would not use math mode tex, i.e. no $ signs around the text, and therefore would use the textfont specified in the preamble (in this case the font without serifs).

import matplotlib as mpl from matplotlib.ticker import LogFormatterExponent  mpl.use('pgf') pgf_with_latex = {         "pgf.texsystem": "pdflatex",         "font.family": "sans-serif",         "text.usetex": False,         "pgf.preamble": [             r"\usepackage[utf8x]{inputenc}",             r"\usepackage{tgheros}",  # TeX Gyre Heros sans serif             r"\usepackage[T1]{fontenc}"             ]         } mpl.rcParams.update(pgf_with_latex) import matplotlib.pyplot as plt  class LogFormatterTexTextMode(LogFormatterExponent):     def __call__(self, x, pos=None):         x = LogFormatterExponent.__call__(self, x,pos)         s = r"10\textsuperscript{{{}}}".format(x)         return s  fig = plt.figure(figsize=[3, 2]) ax = fig.add_subplot(111) ax.set_yscale("log") ax.yaxis.set_major_formatter(LogFormatterTexTextMode()) ax.minorticks_off() ax.set_xlabel("sans-serif font label") ax.set_ylabel("text mode tex label") plt.gca().set_ylim([0.01, 20000]) plt.gcf().tight_layout()   plt.savefig('{}.pdf'.format("test")) 

Answers 2

You can define your own FuncFormatter that does the scientific notation in unicode.

An almost complete converter to superscript was given in this answer. I just added the minus.

Here's an implementation:

# -*- coding: utf-8 -*- from math import log10  SUPERSCRIPTS = dict(zip(u"-0123456789", u"⁻⁰¹²³⁴⁵⁶⁷⁸⁹")) def unicode_sci_notation(x, pos):     """Scientific notation of number with unicode"""     power = int(log10(x))     mantissa = x/(10**power)     superscript = u''.join(SUPERSCRIPTS[c] for c in unicode(power))     if mantissa == 1:         return '10%s' % superscript     else:         return '%.2f x 10%s' % (mantissa, superscript) formatter = mpl.ticker.FuncFormatter(unicode_sci_notation)  ax.yaxis.set_major_formatter(formatter) 

To do it this way, you need to specify coding: utf-8 at the top of your script. If you don't want that, you can escape the unicode characters as explained in the answer I linked.

Read More

GCP Authentication: RefreshError

In order to round-trip test mail sending code in our GCP backend I am sending an email to a GMail inbox and attempting to verify its arrival. The current mechanism for authentication to the GMail API is fairly standard, pasted from the GMail API documentation and embedded in a function:

def authenticate():     """Authenticates to the Gmail API using data in credentials.json,     returning the service instance for use in queries etc."""     store = file.Storage('token.json')     creds = store.get()     if not creds or creds.invalid:         flow = client.flow_from_clientsecrets(CRED_FILE_PATH, SCOPES)         creds = tools.run_flow(flow, store)     service = build('gmail', 'v1', http=creds.authorize(Http()))     return service 

CRED_FILE_PATH points to a downloaded credentials file for the service. The absence of the token.json file triggers its re-creation after an authentication interaction via a browser window, as does the token's expiry.

This is an integration test that must run headless (i.e. with no interaction whatsoever). When re-authentication is required the test currently raises an exception when the authentication flow starts to access sys.argv, which means it sees the arguments to pytest!

I've been trying to find out how to authenticate reliably using a mechanism that does not require user interaction (such as an API key). Nothing in the documentation or on Stackoverflow seems to answer this question.

A more recent effort uses the keyfile from a service account with GMail delegation to avoid the interactive Oauth2 flows.

def authenticate():     """Authenticates to the Gmail API using data in g_suite_access.json,     returning the service instance for use in queries etc."""     main_cred = service_account.Credentials.from_service_account_file(         CRED_FILE_PATH, scopes=SCOPES)     # Establish limited credential to minimise any damage.     credentials = main_cred.with_subject(GMAIL_USER)     service = build('gmail', 'v1', credentials=credentials)     return service 

On trying to use this service with

        response = service.users().messages().list(userId='me',                                     q=f'subject:{subject}').execute() 

I get:

google.auth.exceptions.RefreshError:   ('unauthorized_client: Client is unauthorized to retrieve access tokens using this method.',    '{\n "error": "unauthorized_client",\n "error_description": "Client is unauthorized to retrieve access tokens using this method."\n}') 

I get the feeling there's something fundamental I'm not understanding.

1 Answers

Answers 1

The service account needs to be authorized or it cant access the emails for the domain.

"Client is unauthorized to retrieve access tokens using this method"

Means that you have not authorized it properly; check Delegating domain-wide authority to the service account

Source: Client is unauthorized to retrieve access tokens using this method Gmail API C#

Read More

“ValueError: Trying to share variable $var, but specified dtype float32 and found dtype float64_ref” when trying to use get_variable

I am trying to build a custom variational autoencoder network, where in I'm initializing the decoder weights using the transpose of the weights from the encoder layer, I couldn't find something native to tf.contrib.layers.fully_connected so I used tf.assign instead, here's my code for the layers:

def inference_network(inputs, hidden_units, n_outputs):     """Layer definition for the encoder layer."""     net = inputs     with tf.variable_scope('inference_network', reuse=tf.AUTO_REUSE):         for layer_idx, hidden_dim in enumerate(hidden_units):             net = layers.fully_connected(                 net,                 num_outputs=hidden_dim,                 weights_regularizer=layers.l2_regularizer(training_params.weight_decay),                 scope='inf_layer_{}'.format(layer_idx))             add_layer_summary(net)         z_mean = layers.fully_connected(net, num_outputs=n_outputs, activation_fn=None)         z_log_sigma = layers.fully_connected(             net, num_outputs=n_outputs, activation_fn=None)      return z_mean, z_log_sigma   def generation_network(inputs, decoder_units, n_x):     """Define the decoder network."""     net = inputs  # inputs here is the latent representation.     with tf.variable_scope("generation_network", reuse=tf.AUTO_REUSE):         assert(len(decoder_units) >= 2)         # First layer does not have a regularizer         net = layers.fully_connected(             net,             decoder_units[0],             scope="gen_layer_0",         )         for idx, decoder_unit in enumerate([decoder_units[1], n_x], 1):             net = layers.fully_connected(                 net,                 decoder_unit,                 scope="gen_layer_{}".format(idx),                 weights_regularizer=layers.l2_regularizer(training_params.weight_decay)             )     # Assign the transpose of weights to the respective layers     tf.assign(tf.get_variable("generation_network/gen_layer_1/weights"),               tf.transpose(tf.get_variable("inference_network/inf_layer_1/weights")))     tf.assign(tf.get_variable("generation_network/gen_layer_1/bias"),               tf.get_variable("generation_network/inf_layer_0/bias"))     tf.assign(tf.get_variable("generation_network/gen_layer_2/weights"),               tf.transpose(tf.get_variable("inference_network/inf_layer_0/weights")))     return net # x_recon 

It is wrapped using this tf.slim arg_scope:

def _autoencoder_arg_scope(activation_fn):     """Create an argument scope for the network based on its parameters."""      with slim.arg_scope([layers.fully_connected],                         weights_initializer=layers.xavier_initializer(),                         biases_initializer=tf.initializers.constant(0.0),                         activation_fn=activation_fn) as arg_sc:         return arg_sc 

However I'm getting the error: ValueError: Trying to share variable VarAutoEnc/generation_network/gen_layer_1/weights, but specified dtype float32 and found dtype float64_ref. I have narrowed this down to the get_variablecall, but I don't know why it's failing.

If there is a way where you can initialize a tf.contrib.layers.fully_connected from another fully connected layer without a tf.assign operation, that solution is fine with me.

1 Answers

Answers 1

I can't reproduce your error. Here is a minimalistic runnable example that does the same as your code:

import tensorflow as tf  with tf.contrib.slim.arg_scope([tf.contrib.layers.fully_connected],                                weights_initializer=tf.contrib.layers.xavier_initializer(),                                biases_initializer=tf.initializers.constant(0.0)):    i = tf.placeholder(tf.float32, [1, 30])    with tf.variable_scope("inference_network", reuse=tf.AUTO_REUSE):     tf.contrib.layers.fully_connected(i, 30, scope="gen_layer_0")    with tf.variable_scope("generation_network", reuse=tf.AUTO_REUSE):     tf.contrib.layers.fully_connected(i, 30, scope="gen_layer_0",       weights_regularizer=tf.contrib.layers.l2_regularizer(0.01))    with tf.variable_scope("", reuse=tf.AUTO_REUSE):     tf.assign(tf.get_variable("generation_network/gen_layer_0/weights"),               tf.transpose(tf.get_variable("inference_network/gen_layer_0/weights"))) 

The code runs without a ValueError. If you get a ValueError running this, then it is probably a bug that has been fixed in a later tensorflow version (I tested on 1.9). Otherwise the error is part of your code that you don't show in the question.

By the way, assign will return an op that will perform the assignment once the returned op is run in a session. So you will want to return the output of all assign calls in the generation_network function. You can bundle all assign ops into one using tf.group.

Read More

Feature extraction and take color histogram

I am working on an image processing feature extraction. I have a photo of a bird in which I have to extract bird area and tell what color the bird has. I used canny feature extraction method to get the edges of a bird.

How to extract only bird area and make the background to blue color?

openCv solution should also be fine.

import skimage import numpy as np %matplotlib inline import matplotlib.pyplot as plt  import os filename = os.path.join(os.getcwd(),'image\image_bird.jpeg') from skimage import io bird =io.imread(filename,as_grey=True) plt.imshow(bird) 

from skimage import feature edges = feature.canny(bird,sigma=1) plt.imshow(edges ) 

Actual bird image can be taken from bird link

2 Answers

Answers 1

1. Identify the edges of your image

2. Binarize the image via automatic thresholding

3. Use contour detection to identify black regions which are inside a white region and merge them with the white region. (Mockup, image may slightly vary)

4. Use the created image as mask to color the background and color it This can be done by simply setting each background pixel (black) to its respective color.

As you can see, the approach is far from perfect, but should give you a general idea about how to accomplish your task. The final image quality might be improved by slightly eroding the map to tighten it to the contours of the bird. You then also use the mask to calculate your color histogram by only taking foreground pixels into account. Edit: Look here:

1. Eroded mask

2. Final image

Answers 2

According to this article https://www.pyimagesearch.com/2016/04/11/finding-extreme-points-in-contours-with-opencv/ and this question CV - Extract differences between two images

I wrote some python code as below. As my predecessor said it is also far from perfect. The main disadvantages of this code are constants value to set manually: minThres (50), maxThres(100), dilate iteration count and erode iteration count.

import cv2 import numpy as np  windowName = "Edges" pictureRaw = cv2.imread("bird.jpg")  ## set to gray pictureGray = cv2.cvtColor(pictureRaw,  cv2.COLOR_BGR2GRAY)  ## blur pictureGaussian = cv2.GaussianBlur(pictureGray, (7,7), 0)  ## canny edge detector - you must specify threshold values pictureCanny = cv2.Canny(pictureGaussian, 50, 100)  ## perform a series of erosions + dilations to remove any small regions of noise pictureDilate = cv2.dilate(pictureCanny, None, iterations=20) pictureErode = cv2.erode(pictureDilate, None, iterations=5)  ## find the nozero regions in the erode imask2 = pictureErode>0  ## create a Mat like pictureRaw canvas = np.full_like(pictureRaw, np.array([255,0,0]), dtype=np.uint8)  ## set mask  canvas[imask2] = pictureRaw[imask2] cv2.imwrite("result.png", canvas) 

Read More

Tensorflow Estimator: Cache bottlenecks

When following the tensorflow image classification tutorial, at first it caches the bottleneck of each image:

def: cache_bottlenecks())

I have rewritten the training using tensorflow's Estimator. This really simplified all the code. However I want to cache the bottleneck features here.

Here is my model_fn. I want to cache the results of the dense layer so I can make changes to the actual training without having to compute the bottlenecks each time.

How can I accomplish that?

def model_fn(features, labels, mode, params):     is_training = mode == tf.estimator.ModeKeys.TRAIN      num_classes = len(params['label_vocab'])      module = hub.Module(params['module_spec'], trainable=is_training and params['train_module'])     bottleneck_tensor = module(features['image'])      with tf.name_scope('final_retrain_ops'):         logits = tf.layers.dense(bottleneck_tensor, units=num_classes, trainable=is_training)  # save this?      def train_op_fn(loss):         optimizer = tf.train.AdamOptimizer()         return optimizer.minimize(loss, global_step=tf.train.get_global_step())      head = tf.contrib.estimator.multi_class_head(n_classes=num_classes, label_vocabulary=params['label_vocab'])      return head.create_estimator_spec(         features, mode, logits, labels, train_op_fn=train_op_fn     ) 

0 Answers

Read More

Convert Non-Searchable Pdf to Searchable Pdf in Windows Python

Need a solution to convert a PDF file where every page is image and a page can either contains text, table or combination of both to a searchable pdf.

I have used ABBY FineReader Online which is doing the job perfectly well but I am looking for a solution which can be achieved via Windows Python

I have done detailed analysis and below are the links which came close to what I want but not exactly:

Scanned Image/PDF to Searchable Image/PDF

It is telling to use Ghost script to convert it 1st to image and then it does directly convert to text. I don't believe tesseract converts non-searchable to searchable PDF's.

Converting searchable PDF to a non-searchable PDF

The above solution helps in reverse i.e. converting searchable to non-searchable. Also I think these are valid in Ubuntu/Linux/MacOS.

Can someone please help in telling what should be the Python code for achieving non-searchable to searchable in Windows Python?

---

UPDATE 1

I have got the desired result with Asprise Web Ocr. Below is the link and code:

https://asprise.com/royalty-free-library/python-ocr-api-overview.html

I am looking for a solution which can be done through Windows Python libraries only as

1. Need not to pay subscription costs in future
2. I need to convert thousands of documents daily and it will be cumbersome to upload one to API and then download and so on.

UPDATE 2

I know the solution of converting non-searchable pdf directly to text. But I am looking is their any way to convert non-searchable to searchable PDF. I have the code for converting the PDF to text using PyPDF2.

3 Answers

Answers 1

Well you don't actually need to transform everything inside the pdf to text. Text will remain text, table will remain table and if possible image should become text. You would need a script that actually reads the pdf as is, and begins the conversion on blocks. The script would write blocks of text until the document has been read completely and then transform it into a pdf. Something like

if line_is_text():     write_the_line_as_is() elif line_is_img():     transform_img_in_text()# comments below code ... .. . 

Now transform_img_in_text() I think it could be done with many external libraries, one you can use could be:

Tesseract OCR Python

You can download this lib via pip, instructions provided in the link above.

Answers 2

If an online ocr solution is acceptable to you, the free OCR API from OCR.space can also create searchable PDFs and works well.

In the free version the created PDF contains a watermark. To remove the watermark you need to upgrade to their commercial PRO plan. You can test the api with the web form on the front page.

OCR.space is also available as non-subscription on-premise option, but I am unsure about the price. Personally I use the free ocr api with good success.

Answers 3

I've used pypdfocr in the past to do this. It hasn't been updated recently though.

From the README:

pypdfocr filename.pdf --> filename_ocr.pdf will be generated 

Read carefully the Install instructions for Windows.

Read More

Docker, Flask, SQLAlchemy: ValueError: invalid literal for int() with base 10: 'None'

I have a flask app that can be initialized successfully and connects to Postgresql database. However, when i try to dockerize this app, i get the below error message. "SQLALCHEMY_DATABASE_URI" is correct and i can connect to it, so i can't figure where I have gone wrong.

docker-compose logs

app_1       |   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line 60, in __init__ app_1       |     self.port = int(port) app_1       | ValueError: invalid literal for int() with base 10: 'None' 

Postgres database connects successfully in Docker container

postgres_1  | LOG:  database system is ready to accept connections 

config.py

from os import environ import os  RDS_USERNAME = environ.get('RDS_USERNAME') RDS_PASSWORD = environ.get('RDS_PASSWORD') RDS_HOSTNAME = environ.get('RDS_HOSTNAME') RDS_PORT = environ.get('RDS_PORT') RDS_DB_NAME = environ.get('RDS_DB_NAME')  SQLALCHEMY_DATABASE_URI = "postgresql+psycopg2://{username}:{password}@{hostname}:{port}/{dbname}"\                           .format(username = RDS_USERNAME, password = RDS_PASSWORD, \                            hostname = RDS_HOSTNAME, port = RDS_PORT, dbname = RDS_DB_NAME) 

flask_app.py (entry point)

def create_app():     app = Flask(__name__, static_folder="./static", template_folder="./static")     app.config.from_pyfile('./app/config.py', silent=True)      register_blueprint(app)     register_extension(app)      with app.app_context():         print(db) -> This prints the correct path for SQLALCHEMY_DATABASE_URI         db.create_all()         db.session.commit()     return app  def register_blueprint(app):     app.register_blueprint(view_blueprint)     app.register_blueprint(race_blueprint)   def register_extension(app):     db.init_app(app)     migrate.init_app(app)   app = create_app()  if __name__ == '__main__':     app.run(host='0.0.0.0', port=8080, debug=True) 

Dockerfile

FROM ubuntu  RUN apt-get update && apt-get -y upgrade  RUN apt-get install -y python-pip && pip install --upgrade pip  RUN mkdir /home/ubuntu  WORKDIR /home/ubuntu/celery-scheduler  ADD requirements.txt /home/ubuntu/celery-scheduler/  RUN pip install -r requirements.txt  COPY . /home/ubuntu/celery-scheduler  EXPOSE 5000  CMD ["python", "flask_app.py", "--host", "0.0.0.0"] 

docker-compose.yml

version: '2'   services:   app:     restart: always     build:        context: .       dockerfile: Dockerfile     volumes:       - .:/app     depends_on:       - postgres    postgres:     restart: always       image: postgres:9.6     environment:       - POSTGRES_USER=${RDS_USERNAME}       - POSTGRES_PASSWORD=${RDS_PASSWORD}       - POSTGRES_HOSTNAME=${RDS_HOSTNAME}       - POSTGRES_DB=${RDS_DB_NAME}     ports:       - "5432:5432" 

1 Answers

Answers 1

You need to set environment variables RDS_USERNAME, RDS_PASSWORD, RDS_HOSTNAME, RDS_PORT , and RDS_DB_NAME in Dockerfile with ENV key value, for example

ENV RDS_PORT 5432 

Read More