Showing posts with label database-design. Show all posts
Showing posts with label database-design. Show all posts

Monday, September 10, 2018

Generate dynamic schedule for Rails application

Leave a Comment

I need to implement scheduling task for my application. Let's say application display popup questions fetching the schedule data from database.

Heres the database table structure -enter image description here

Now i want to display question's to logged in user from QuestionSchedule table. Heres the scenario - Question1 should displays X repeatable_times after each X repeat_after_days. Example - Question1 should displays 3 repeatable_times after each 2 repeat_after_days.

Note - UserQuestionAnswer should not display duplicate entry with calculate the UserQuestionAnswer and QuestionSchedule table.

Details data - Question (id-1, title- What is your level of confidence for todays task ?) QuestionSchedule(id-1,question_id-1,repeatable_times-3,repeat_after_days-2) UserQuestionAnswer(id-1,question_id-1,user_id-1,answer_at-(2018-08-27))

Now i want to generate schedule on fly -

2018-08-25 -> Schedule is created

2018-08-26 -> Should not display

2018-08-27 -> Should display and add answer to UserQuestionAnswer table not twice

2018-08-28 -> Should not display

2018-08-29 -> Should display and add answer to UserQuestionAnswer table not twice

2018-08-30 -> Should not display

2018-08-31 -> Should display and add answer to UserQuestionAnswer table not twice

0 Answers

Read More

Tuesday, May 29, 2018

Database Design and Query for Historical Tabular Data

Leave a Comment

I have a set of HTML tables that store survey questions and responses over time. Each question has it's own HTML table, the columns are the years, the rows are the responses, then the individual cells have the number of responses for that year, as shown below:

enter image description here

I've gone back and forth on how to normalize this data and store it in a database, but I'm not sure what the best way is. I'm looking for a good database schema that can handle additional questions, responses, and years as time goes by. I'm also looking for a good query that can output an HTML table like below. I can do it easily enough in a PHP loop, but I'm worried that isn't good for performance.

Right now, I have the following table design:

question

id int(11) unsigned AI PK name varchar(255) UNQ number varchar(255) UNQ text

year

id int(11) unsigned AI PK question_id int(11) unsigned FK name varchar(255) UNQ (question_id + name)

response

id int(11) unsigned AI PK question_id int(11) unsigned FK name varchar(255) UNQ (question_id + name)

data

id int(11) unsigned AI PK question_id int(11) unsigned FK year_id int(11) unsigned FK response_id int(11) unsigned FK UNQ (year_id + response_id) count int(11) unsigned NULL

Any help or improvements would be greatly appreciated.

3 Answers

Answers 1

You don't need the table year, since a year is question-independently.

And alter table data

  • year_id int(11) unsigned FK to year SMALLINT(4) UNSIGNED

  • UNQ (year_id + response_id) to UNQ (year + response_id)

Answers 2

In general, if you have a UNIQUE key (is that what 'UNQ' means??), use if for the PRIMARY KEY.

"names" usually don't need to be VARCHAR(255). Pick a smaller size.

"numbers" usually don't need to be VARCHAR(255). Pick a more appropriate datatype.

Write you schema in CREATE TABLE syntax; I am having severe trouble parsing your run-on description.

What does "0.00" represent? Is it derivable from the other data? If so, do not store it.

From the second images provide, I would guess you have 1 table:

CREATE TABLE foo (     year YEAR NULL,     gender ENUM('male', 'female') NOT NULL,     val SMALLINT UNSIGNED NOT NULL,     PRIMARY KEY(year, gender) ) ENGINE=InnoDB; 

I don't understand the meaning of '1959-1974', but it might be

    cohort VARHAR(20) NOT NULL 

and replace gender in a second table that otherwise looks like the above table.

But... You can't really design a schema without understanding what will be done with the data. Do you have any tentative SELECTs?

Answers 3

there's good ideas already for a structured version your target data model - if you wanted the structure of your stats to be a little more flexible, but still be able to key and group over time, then an alternative might be to follow bi/ dw pattern to model your data

the following are 'logical' and would correlate with the attributes/ dimensions in a fact table per kimball et. al., where the 'grain' of the fact table is 'src html file + table + row + cell + value(s)', assuming your values are consistent over time

  • (i notice in your image that there are a couple tables for the one html file, and a couple of values in each cell)

  • group_srcfile (points to the location in the source html file/ table/ row/ cell, and you could probably store source html as well, in case you need to perform an autopsy later)

  • group_cohort (points to the normalised cohort eg. '18-24 yr olds over time', or 'males over time')

  • group_question (points to question definition - this is all the same questions over time)

  • question_id (question definition + question year)

  • question_year (this is the year that the question was asked)

  • cohort_start_year (this is the start-year of the cohort was asked the question)

  • cohort_end_year (this is the end-year of the cohort was asked the question)

  • cohort_start_age (if applicable, would be the normalised 'xxxx - yyyy', eg: '18')

  • cohort_end_age (this is either specified, or inferred by a 'xxxx - present' where 'present' is the year of the report html file)

  • values 1 .. n must count the same things, otherwise you would need to split these off as well

to generate decent output you would need to finalise the questions on your data table, but whatever you do, it would be relatively straightforward to export html using php

i thought about the method by which you load the data into mysql, but without solid samples of the html files which are the source of your data, it's hard to write specific code (ie. open in your browser and 'view source', or equivalent)

as a general approach i would parse each fact (table cell td) from the html using php and DOMDocument, then emit a row in denormalised form, for subsequent loading into a staging table and ultimately your fact table

in this context, 'emit' is the source of what ultimately becomes an individual row in your fact table but you can't load it yet because you don't know what your dimension keys will be unless you define them at time of parsing the html

this is virtually impossible to do: instead, load into a loosely defined table (without any ref. integrity) and once you've finished parsing all files, write etl or queries that will generate your dimension tables before finishing up with your facts

(i would probably use pentaho data integration to handle the second phase - its streaming xml parser couldn't handle the first: too strict)


i found this test html file which was sufficiently aged, as to cause me to go and puke my last coffee at the mere thought of endlessly re-writing scraping code to account for the never-ending layout changes courtesy of 'dreamtheaver' ...

once my hands had stabilised enough, and blood-flow had returned to normal, i channeled the machine spirit and produced the following php - notably absent is any sort of restructuring/ denormalising of the source table:

<?php  ini_set('display_errors', 1); ini_set('display_startup_errors', 1); error_reporting(E_ALL);  $dom = new DOMDocument();  $srcfile = 'testxmlparser.html'; $dom->loadHTMLFile( $srcfile, NULL );  echo 'odom is: ' . ($dom ? 'nice':'naughty') . PHP_EOL;  if( $dom ) {   // get all the table rows in the document   $tblrows = $dom->getElementsByTagName('tr');    foreach( $tblrows as $trrow ) {     $tblcells = $trrow->getElementsByTagName('td');     $incr = 0;      // buffer this table row's cell (td) data that we encounter, in case it is interesting...     $srowbuf = '';     foreach( $tblcells as $tdcell ) {       $srowbuf = ($srowbuf . $tdcell->nodeValue);       if( 1 <= $incr++ )         $srowbuf = ($srowbuf . '+|');     }     // we know the table data we're interested in has 12 cells only     if( 12 == $incr )       echo $srowbuf . '+|' . $incr . '+|' . $srcfile . PHP_EOL;   } }  ?> 
Read More

Saturday, May 5, 2018

How to properly store the necessary info to register the user and other participants in the congress? (scenario 1 works but scenario 2 and 3 dont)

Leave a Comment

I have a congress registration page that has the form below that the user should fill to register in the congress. In this registration form:

  • If "all_participants" column is "1" in the congress table, it appears for each selected ticket in the previous page (congress details page) a section for the user that is doing the registration insert the name and surname of each participant (that is, associating each selected ticket in the previous page to a participant name and surname)

  • If the all_participants column is "0", it appears the "<p>Is not necessary additional info.</p>" because is not necessary to collect info of each participant (for each selected ticket). The registration is done using the information of the authenticated user.

  • the selected tickets from the previous page are available in the variable "$selectedTypes"

// registration form in the registration.blade.php page

      <form method="post" id="step1form" action="">     {{csrf_field()}}     @if (!empty($allParticipants))         @if($allParticipants == 1)             <p>Please fill in all fields. Your tickets will be sent to                 p{{ (\Auth::check()) ? Auth::user()->email : old('email')}}.</p>              @foreach($selectedTypes as $selectedType)                 @foreach(range(1,$selectedType['quantity']) as $test)                      <h6>Participant - 1 - {{$test}}</h6>                     <div class="form-check">                         <input class="form-check-input" type="radio" name="payment_method" value="referencias">                         <label class="form-check-label d-flex align-items-center" for="exampleRadios1">                             <span class="mr-auto">Fill the following fields with the authenticated user information.</span>                         </label>                     </div>                     <div class="form-group font-size-sm">                         <label for="participant_name" class="text-gray">Name</label>                         <input type="text" name="participant_name[]" required class="form-control" value="">                     </div>                     <div class="form-group font-size-sm">                         <label for="participant_surname" class="text-gray">Surname</label>                         <input type="text" required class="form-control" name="participant_surname[]" value="">                     </div>                     <input type="hidden" name="ttypes[]" value="{{ $selectedType['id'] }}"/>                     @foreach($selectedType['questions'] as $customQuestion)                         <div class="form-group">                             <label for="participant_question">{{$customQuestion->question}}</label>                             <input type="text"                                    @if($customQuestion->pivot->required == "1") required @endif                                    class="form-control" name="participant_question[]">                             <input type="hidden" name="participant_question_required[]"                                    value="{{ $customQuestion->pivot->required }}">                             <input type="hidden" value="{{ $customQuestion->id }}" name="participant_question_id[]"/>                         </div>                     @endforeach                 @endforeach             @endforeach             @else                 <p>Its not necessary aditional info. Your tickets will be sent to {{ (\Auth::check()) ? Auth::user()->email : old('email')}}.</p>              @endif         @endif      <input type="submit" href="#step2"            id="goToStep2Free" class="btn btn-primary btn float-right next-step" value="Go to step 2"/> </form> 

Issues: For the user register in the conference can exist 3 scenarios. The first scenario is working fine, Im getting issues in the scenario 2 and 3.

Scenario 1 is working fine: "all_participants" column is "1" in the congresses table which means that is necessary to collect info of each participant and the user selected in the previous page (congress details page) 1 or more ticket types that have asociated one or more custom questions. This is the only scenario that is working fine. Diagram demonstrating the scenario:

enter image description here

This is working fine. The registrations and participants table stay like:

Registration table

id  congress_id     main_participant_id 1        1                   1            (user 1 registers in the congress with id 1) 

Participants table

id     registration_id     ticket_type_id       name     surname 1            1                     1             John       X 2            1                     2             Jake       Y 

Answers table

id     participant_id     question_id       answer      1            1                     1          000        

Questions table

id           question                        congress_id       answer      1          Whats your phone?                    1              000        

Scenario 2 is not working fine The scenario 2 is If all_participants is "1" and the user select ticket types that dont have any no custom questions associated. In this scenario Im getting this error below because "$request->participant_question_required" dont exist:

 "Invalid argument supplied for foreach()" in          ...         else {             $messages = [                 'participant_question.*.required' => 'The participant is required'             ];             foreach ($request->participant_question_required as $key => $value) {         ... 

I can put "if(isset($request->participant_question_required)) {" and then all code inside of this, but like that if the congress dont have any custom question associated then dont appears any error but no record are inserted in the registrations and participants table.

Diagram of the scenario 2 to demonstrate the issue:

enter image description here

Scenario 3 is not working fine: If all_participants is "0" and the user select ticket types that have 1 or more custom questions associated, in the registartion form dont appears any custom question. But if there are custom questions for some ticket type selected by the user and "all_participant" is "0" which means that is only necessary to collect info from the user that is doing the registration (the authenticated user) it shoud appear the custom question(s) associated to the selected ticket type(s) for the user that is doing the registration to answer.

Diagram of the scenario 3 to demonstrate the issue:

enter image description here

// complete method to register the user in the congress

public function StoreUserInfo(Request $request, $id, $slug = null, Validator $validator){     $allParticipants = Congress::where('id', $id)->first()->all_participants;     $user = Auth::user();      if($allParticipants){         $rules = [             'participant_name.*' => 'required|max:255|string',             'participant_surname.*' => 'required|max:255|string',         ];          $messages = [             'participant_question.*.required' => 'The participant is required'         ];          foreach ($request->participant_question_required as $key => $value) {             $rule = 'string|max:255'; // I think string should come before max             //dd($value);             // if this was required, ie 1, prepend "required|" to the rule             if ($value) {                 $rule = 'required|' . $rule;             }              // add the individual rule for this array key to the $rules array             $rules["participant_question.{$key}"] = $rule;         }          $validator = Validator::make($request->all(), $rules, $messages);          if($validator->passes()) {             $registration = Registration::create([                 'congress_id' => $id,                 'main_participant_id' => $user->id,                 'status' => 'C',             ]);              $participants = [];              for ($i = 0; $i < count($request->participant_name); $i++)                 $participants[] = Participant::create([                     'name' => $request->participant_name[$i],                     'surname' => $request->participant_surname[$i],                     'registration_id' => $registration->id,                     'ticket_type_id' => $request->rtypes[$i]                  ]);              for ($i = 0; $i < count($request->participant_question); $i++)                 $answer = Answer::create([                     'question_id' => $request->participant_question_id[$i],                     'participant_id' => $participants[$i]->id,                     'answer' => $request->participant_question[$i],                 ]);             }          return response()->json([             'success' => true,             'message' => 'success'         ], 200);     }       else {          $messages = [             'participant_question.*.required' => 'The participant is required'         ];           foreach ($request->participant_question_required as $key => $value) {             $rule = 'string|max:255'; // I think string should come before max             //dd($value);             // if this was required, ie 1, prepend "required|" to the rule             if ($value) {                 $rule = 'required|' . $rule;             }              // add the individual rule for this array key to the $rules array             $rules["participant_question.{$key}"] = $rule;         }           $validator = Validator::make($request->all(), $rules, $messages);           if ($validator->passes()) {              $registration = Registration::create([                 'congress_id' => $id,                 'main_participant_id' => $user->id,                 'status' => 'C',              ]);              $participants = [];              for ($i = 0; $i < count($request->participant_name); $i++)                 $participants[] = Participant::create([                     'name' => '',                     'surname' => '',                     'registration_id' => $registration->id,                     'ticket_type_id' => $request->rtypes[$i]                  ]);              for ($i = 0; $i < count($request->participant_question); $i++)                 $answer = Answer::create([                     'question_id' => $request->participant_question_id[$i],                     'participant_id' => $participants[$i]->id,                     'answer' => $request->participant_question[$i],                 ]);         }          return response()->json([             'success' => true,             'message' => 'success'         ], 200);      } } 

Relevant models to the question:

class Congress extends Model {     // A conference has many ticket types     public function ticketTypes(){         return $this->hasMany('App\TicketType', 'congress_id');     }      public function registrations(){         return $this->hasMany('App\Registration', 'congress_id');     } }  // RegistrationModel class Registration extends Model {     // a registration has one user that do the registration (main_participant_id)     public function customer(){         return $this->belongsTo('App\User');     }     public function congress(){         return $this->belongsTo('App\Congress');     } }  class TicketType extends Model {     public function congress(){         return $this->belongsTo('App\Congress');     } }  class Question extends Model {     public function registration_type(){         return $this->belongsToMany('App\RegistrationType', 'ticket_type_questions')             ->withPivot('required');     } }  class Answer extends Model {     public function question(){         return $this->belongsTo('Question');     }     public function participant(){         return $this->belongsTo('Participant');     } } 

3 Answers

Answers 1

If you wish to store more than one ticket_type to a participant you will need to use a pivot table. A pivot table is a table that only contains ids and relates data together in a many to many fashion.

e.g.

participants_ticket_type table id      ticket_type_id   participant_id 1          1               2 2          2               2 

Notice how participant 2 has ticket_types of 1 and 2? This is a many to many relationship.

Laravel handles these relationships for you and you can find it in their docs. https://laravel.com/docs/5.6/eloquent-relationships#many-to-many

Usually if you find yourself storing more than 1 id in a column a many to many relationship will solve this issue.

Answers 2

I see two things

1.- The form doesn't send anything when $allParticipants == 0, maybe something more like:

    <form method="post" id="step1form" action="">     {{csrf_field()}}     @if (!is_null($allParticipants) && is_int($allParticipants))         @if($allParticipants == 1)                <p>Please fill in all fields. Your tickets will be sent to                 p{{ (\Auth::check()) ? Auth::user()->email : old('email')}}.</p>         @else             <p>Its not necessary aditional info. Your tickets will be sent to {{ (\Auth::check()) ? Auth::user()->email : old('email')}}.</p>         @endif          @foreach($selectedTypes as $selectedType)             @foreach(range(1,$selectedType['quantity']) as $test)                 <h6>Participant - 1 - {{$test}}</h6>                 <div class="form-check">                     <input class="form-check-input" type="radio" name="payment_method" value="referencias">                     <label class="form-check-label d-flex align-items-center" for="exampleRadios1">                         <span class="mr-auto">Fill the following fields with the authenticated user information.</span>                     </label>                 </div>                 @if($allParticipants == 1)                     <div class="form-group font-size-sm">                         <label for="participant_name" class="text-gray">Name</label>                         <input type="text" name="participant_name[]" required class="form-control" value="">                     </div>                     <div class="form-group font-size-sm">                         <label for="participant_surname" class="text-gray">Surname</label>                         <input type="text" required class="form-control" name="participant_surname[]" value="">                     </div>                    @foreach($selectedType['questions'] as $customQuestion)                     <div class="form-group">                         <label for="participant_question">{{$customQuestion->question}}</label>                         <input type="text"                                 @if($customQuestion->pivot->required == "1") required @endif                                 class="form-control" name="participant_question[]">                         <input type="hidden" name="participant_question_required[]"                                 value="{{ $customQuestion->pivot->required }}">                         <input type="hidden" value="{{ $customQuestion->id }}" name="participant_question_id[]"/>                     </div>                    @endforeach                 @else                     <input type="hidden" value="foo" name="participant_name[]"/>                     <input type="hidden" value="bar" name="participant_surname[]"/>                 @endif                 <input type="hidden" name="ttypes[]" value="{{ $selectedType['id'] }}"/>                             @endforeach             @if ($allParticipants == 0)                @foreach($selectedType['questions'] as $customQuestion)                     <div class="form-group">                         <label for="participant_question">{{$customQuestion->question}}</label>                         <input type="text"                                 @if($customQuestion->pivot->required == "1") required @endif                                 class="form-control" name="participant_question[]">                         <input type="hidden" name="participant_question_required[]"                                 value="{{ $customQuestion->pivot->required }}">                         <input type="hidden" value="{{ $customQuestion->id }}" name="participant_question_id[]"/>                     </div>                 @endforeach             @endif         @endforeach     @endif      <input type="submit" href="#step2"             id="goToStep2Free" class="btn btn-primary btn float-right next-step" value="Go to step 2"/>     </form> 

2.- Questions seems to depend on Ticket types and are optionals, the function should take it into account.

public function StoreUserInfo(Request $request, $id, $slug = null, Validator $validator){     $allParticipants = Congress::where('id', $id)->first()->all_participants;     $user = Auth::user();      $rules = [];     $messages = [];      if(isset($request->participant_question_required)) {         $messages = [             'participant_question.*.required' => 'The participant is required'         ];          foreach ($request->participant_question_required as $key => $value) {             $rule = 'string|max:255'; // I think string should come before max             //dd($value);             // if this was required, ie 1, prepend "required|" to the rule             if ($value) {                 $rule = 'required|' . $rule;             }              // add the individual rule for this array key to the $rules array             $rules["participant_question.{$key}"] = $rule;         }     }      if($allParticipants){         $rules = [             'participant_name.*' => 'required|max:255|string',             'participant_surname.*' => 'required|max:255|string',         ];     }      $validator = Validator::make($request->all(), $rules, $messages);      if($validator->passes()) {         $registration = Registration::create([             'congress_id' => $id,             'main_participant_id' => $user->id,             'status' => 'C',         ]);          $participants = [];         for ($i = 0; $i < count($request->participant_name); $i++) {             $name = ($allParticipants) ? $request->participant_name[$i] : '';             $surname = ($allParticipants) ? $request->participant_surname[$i] : '';             $participants[] = Participant::create([                 'name' => $name,                 'surname' => $surname,                 'registration_id' => $registration->id,                 'ticket_type_id' => $request->rtypes[$i]              ]);         }          if (isset($request->participant_question))             for ($i = 0; $i < count($request->participant_question); $i++)                     $answer = Answer::create([                         'question_id' => $request->participant_question_id[$i],                         'participant_id' => $participants[$i]->id,                         'answer' => $request->participant_question[$i],                     ]);     }      return response()->json([         'success' => true,         'message' => 'success'     ], 200); } 

Hope it works! Regards

Answers 3

First of all use select * from TABLENAME and then use the COUNT function either in the query Or use PHP Count function on the laravel blade file then execute your code..

on blade file

@if(count($records)>0){    // Your Code  } 
Read More

Sunday, May 21, 2017

Database performance: Using one entity/table with the max. possible properties or split to different entities/tables?

Leave a Comment

im need to design some database tables but im not sure about the performance impact. In my case its more about the read performance than for saving the data.

The situation

With the help of pattern recognition im finding out how many values of a certain object needs to be saved in my postgresql database. Amount other lets say fixed properties the only difference is if 1, 2 or 3 values of the same type needs to be saved.

Currently im having 3 entities/tables which differ only in having having 1, 2 or 3 not nullable properties of the same type.

For example:

EntityTestOne/TableOne {     ... other (same) properties     String optionOne; }  EntityTestTwo/TableTwo {     ... other (same) properties     String optionOne;     String optionTwo;  }  EntityTestThree/TableThree {     ... other (same) properties     String optionOne;     String optionTwo;     String optionThree; } 

I expect to have several million records in production and im thinking what could be the performance impact of this variant and what could be alternatives.

Alternatives

Other options which come into my mind:

  • Use only one entity class or table with 3 options (optionTwo and optionThree will be nullable then). If to talk of millions of expected records plus caching im asking myself isn't it a kind of 'waste' to save millions of null values in at least two (caching) layers (database itself and hibernate). In a another answer i read yesterday saving a null value in postgresql need only 1 bit what i think isnt that much if we talk about several millions of records which can contain some nullable properties (link).
  • Create another entity/table and use a collection (list or set) relationship instead

For example:

EntityOption {     String value; }  EntityTest {     ... other (same) properties     List<EntityOption> options; } 
  • If to use this relationship: What would give a better performance in case of creating new records: Creating for every new EntityTest new EntityOption's or doing a lookup before and reference a existing EntityOption if exists? What about the read performance while fetching them later and the joins which will be needed then? Compared to the variant with one plain Entity with three options i can imagine it could be slightly slower...

As im not that strong in database design and working with hibernate im interested of the pros and cons of these approaches and if there are even more alternatives. I even would like to ask the question if postgresql is the right choice for this or if should think about using another (free) database.

Thanks!

2 Answers

Answers 1

The case is pretty clear in my opinion: If you have an upper limit of three properties per object, use a single table with nullable attributes.

A NULL value does not take up any space in the database. For every row, PostgreSQL stores a bitmap that contains which attributes are NULL. This bitmap is always stored, except when all attributes are not nullable. See the documentation for details.
So don't worry about storage space in this case.

Using three different tables or storing the attributes in a separate table will probably lead to UNIONs or JOINs in your queries, which will make the queries more complicated and slow.

Answers 2

There are many inheritance strategy for creating entity class, I think you should go with single table strategy, where there will be a discriminator column (managed by hibernate itself), and all common filed will be used by each entity and some specific fields will be use by specific entity and remain null for other entity. This will get improved read performance. For your ref. : http://www.thejavageek.com/2014/05/14/jpa-single-table-inheritance-example/

Read More

Friday, April 7, 2017

How to keep two databases with different schemas up-to-date

Leave a Comment

Our company has really old legacy system with such a bad database design (no foreign keys, columns with serialized PHP arrays, etc. :(). We decided to rewrite a system from a scratch with new database schema.

We want to rewrite a system by parts. So we will split old monolithic application to many smaller ones.

Problem is: we want to have live data in two databases. Old and New schema. I'd like to ask you if anyone of you knows best practices how to do this.

What we think of:

  1. asynchronous data synchronization with message queue
  2. make a REST API in new system and make legacy application to use it instead of db calls
  3. some kind of table replication

Thank you very much

2 Answers

Answers 1

I had to deal with a similar problem in the past. There was a system which didn't have support but there was people using it because, It had some features (security holes) which allowed them certain functionalities. However, they also needed new functionalities.

I selected the tables which involved the new system and I created some triggers for cross update the tables, so when I created a register on the old system the trigger created a copy in the new system and reversal. If you design this system properly you would have both systems working at the same time in real time.

The drawback is that while the both system are running the system is going to become slower since you have to maintain the integrity of two databases in every operation.

Answers 2

I would start by adding a database layer to accept API calls from the business layer, then write to both the old schema and the new. This adds complexity up front, but it lets you guarantee that the data stays in sync.

This would require changing the legacy system to call an API instead of issuing SQL statements. If they did not have the foresight to do that originally, you may not be able to take my approach. But, you should do it going forward.

Triggers may or may not work out. In older versions of MySQL, there can be only one trigger of a given type on a given table. This forces you to lump unrelated things into a single trigger.

Replication can solve some changes -- Engine, datatypes, etc. But it cannot help with splitting one table into two. Be careful of the replication of Triggers and where the Trigger has effect (between Master and Slave). In general, a stored routine should be performed on the Master, letting the effect be replicated to the slave. But it may be worth considering how to have the trigger run in the Slave instead. Or different triggers in the two servers.

Another thought is to do the transformation in stages. By careful planning of schema changes versus application of triggers versus code changes versus database layer, you can do partial transformations one at a time, sometimes without having a big outage to update everything simultaneously (with your fingers crossed). A simple example: (1) change code to dynamically handle either new or old schema, (2) change the schema, (3) clean up the code (remove handling of old schema).

Read More

Wednesday, April 5, 2017

Dealing with disassociated records in a poorly designed database

Leave a Comment

Overview

I have inherited a website that allows users to order customised products. The customisations were saved in a way that disassociates them from their record. I would like to modify the db so these records can be associated.

Example

Users can get Product #1 "stock", or customise it, changing as many as 10 different properties. Let's say color, fabric, width, height etc.

Orders can, and regularly do, contain multiple products, each of which may be customised.

Additionally, users can save their orders, so they can re-order later.

When the database was designed, the details of the order was neatly organised into individual columns. Customer name, address, payment type etc. But the list of products and more notably their customisations were saved as a JSON string in a single column. For ease, let's call this column the "cart".

Basically, the order table has a column cart and the cart column contains a JSON-formatted list of products and customisations.

Unfortunately, the JSON object has reference ids to the product table, but lacks references to the customisation table. Instead it uses a bunch of strings meant for a human to read. Fortunately those strings exist in the customisation table, but they were written as the cart was created.

The problem we face is that the list of customisations can be changed by a CMS. So far, they haven't been changed. 🙏 But they will need to be soon and that's going to cause problems:

Problems

  1. If a customisation option is removed (say, a fabric option) and a customer re-orders from an old saved order, we need to be able to parse the cart, detect this and warn them of the change.

  2. Customisations are currently immutable. Once a product is added to the cart, it cannot be changed. Users need to delete and re-add to make a single change. Poor UX.

  3. If anyone changes the human-readable text on a customisation we're dead. ☠️

Questions

  • How would you design this if you were staring from scratch?

  • How might we go about converting the current implementation and legacy data to this new schema?

I don't know if stack is notable, but we're on Postgres and Django-Python.

2 Answers

Answers 1

I would implement this with the following tables:

Products {   productId                   // primary key   name   price }  Customization_Types {   customizationTypeId         // primary key   name                        // e.g. COLOR, FABRIC, LENGTH }  Customizations {   customizationId             // primary key   customizationTypeId         // foreign key   value                       // e.g. BEIGE, VELVET, 8 }  Product_Customizations {   productCustomizationId      // primary key   productId                   // foreign key   customizationId             // foreign key   priceModifier               // price markup for applying the customization   isValid                     // false if this record is invalid/obsolete }  Orders {   orderId                     // primary key   customerId                  // foreign key }  Product_Orders {   productOrderId              // primary key   orderId                     // foreign key   productId                   // foreign key   quantity }  Customization_Orders {   customizationOrderId        // primary key   productOrderId              // foreign key   productCustomizationId      // foreign key } 

The Products table contains the data for your base products - name, price, etc

The Customization_Types table contains the type names for your different customizations - COLOR, FABRIC, LENGTH, etc

The Customizations table contains a link to a customizationTypeId as well as a legal value - I'm assuming that users can't enter arbitrary numerical values (for e.g. LENGTH or WIDTH) i.e. they're given a drop-down box instead of a text box, however if they can enter arbitrary numerical data then you'll need MIN/MAX fields that are null for named constraints (so e.g. you could have Type:COLOR/Value:BEIGE/Min:NULL/Max:NULL or Type:LENGTH/Value:NULL/Min:4/Max:8)

The Product_Customizations table links a Customization to a Product, so for example if ProductX can come in BEIGE then you would create a Product_Customization record that links ProductX to BEIGE.

The Orders table just contains an orderId and anything else relevant to the order (e.g. a link to the customerId and shippingAddressId)

Product_Orders links a product to an order

Customization_Orders links a Product_Customization to a Product_Order


Let's say a customer orders ProductX in BEIGE and LENGTH=8, then you would create an Order record, a Product_Order record with a link to ProductX, and two Customization_Order records - one linked to COLOR=BEIGE and one linked to LENGTH=8.

This should make it easy to modify a product's customizations without having to reload the entire product - the user can modify color to COLOR=RED without touching the length customization (delete the old Customization_Order:COLOR=BEIGE record and create a new COLOR=RED record), or the user can remove the length customization without touching the color customization (delete the old Customization_Order:LENGTH=8 record).

When reloading an old order/product you can quickly verify that the same productCustomizationIds still apply to the product in question, else flag the user. Additionally, you can flag the user if the customization still applies but the customization's price modifier has changed.


As far as converting the legacy data, I'm not familiar with Python but I do have experience with reading JSON via Java and I'm assuming that Python offers similar if not better libraries for this. The trick is going to be matching the existing data to pre-loaded Product_Customization data - if the data fails to match then create a new Product_Customization row corresponding to it with isValid=FALSE (this is assuming that the customization in question is no longer offered), and when you get a chance manually iterate through the invalid Product_Customization rows to ensure that these really are unmatched customizations and not just parsing errors.

Answers 2

Little improvement to Zim-Zam's answer.

Even better approach is to store not plain values (BEIGE, VELVET, 8) as customization parameters, but kind of schema from which code can build up correct view of a customization.

It could be just JSON/XML formatted text. And the entity that is responsible for building view and applying logic should be able to work with JSON data of different versions.

For example, if properties of a customization have changed and something new has been added, in that case you only need to change code and adjusted JSON will be saved. No need to change existing data. Also there should be possibility to read old JSON versions with old properties and work with it.

Two possible ways of what to do if you read an old entity from DB:

  1. View builder will ignore all old properties of a customization, add new properties and set their values to default. I would go with that personally.
  2. Old view is presented to user, but when user clicks, for example, Ok button or Finish, additional logic will check that there are old properties and notifies user that they should be removed manually or just removes them automatically.

More flexible approach that requires only code changes without touching db and allows to show user old customization properties if it is necessary.

Update: Customizations could have two kind of properties: one that administrator can define, such as title or price, which are not frequently changed and common for all customizations and another one such as size and color which could be changed frequently, could have user defined values and are not common for all customizations.

The first kind should be stored in Customization table as separate columns. That will allow to changed such properties in administrative panel and have all previously stored data consistent.

The second kind of properties could be 1) frequently changed 2) not all customization types could have such properties. It is a bad idea to store them as separate columns because if there are huge amount of data, changing column type or adding new column could cause performance degradation and sometimes could not be possible due to incompatible types of properties.
Actually, if they are stored as separate columns, you are probably will have to change code to support new properties anyway.

My idea is that you still allow administrator to change type of such properties and add new one or remove old one through some interface. The key thing here is that you are storing JSON data like this

{     "properties": {             {                 "propertyName": "height",                 "propertyType": "int",                 "min" : 10,                 "max" : 25,             },             {                 "propertyName": "color",                 "propertyType": "color",             },             {                 "propertyName": "anotherCustomField",                 "propertyType": "text",             },         } } 

What's left to do is to implement view builders or renderers for all kinds of property type. And add a separate table with only values. You fetched a customization record from db, you found out which customization properties there are, checked which one are still valid and rendered only valid ones. If administrator changed type of customization's property or just removed one, you marked that customization's property as not valid in db and that's all the work. No code changes, no database schema changes.

Read More

Monday, April 11, 2016

Database design for classified ad item specification

Leave a Comment

I'm working on a classified ads site with 12 categories. E.g. category vehicles has items cars, bikes, Commercial Vehicles and spare parts. The following is a flow diagram for posting an ad:

When user want to post an ad

I need to show the specification in the Form Filled section of the above image to the users in dropdown lists in the form when they are posting an advertisement. The car specification will be its color,engine,fuel type.

The ERD is below :

ERD

How should this issue be tackled, what are the best practices and is the current design going along the right lines?

4 Answers

Answers 1

On the whole this looks ok. Here are some observations:

  1. likes.iker_id should point at users.id? Just trying to understand your model to start.
  2. I would probably change the pics table to be one pic per row and then add an ordinal for ordering.
  3. One question here is how you intend to look at your graph model. As it is, you might have a graph that could be traversed easily to a depth, a couple deep. I assume you are doing this to recommend ads. If so, I think this is sufficient. If not it would be good to further discuss which rdbms you are targetting.

Answers 2

Hope this helps:
In a simplified case, you will need some extra tables.

enter image description here

Answers 3

So, you are trying to be able to have different specifications for different items in your categories? Or, in other words, it is like having different attributes for different types of products in an e-commerce website.

If that the problem you are tackling, then you should look into the Entity–Attribute–Value (EAV) model that is how the problem is solved. By the way, one of the most popular open source e-commerce engines uses it as well.

enter image description here

Answers 4

i agree look at EAV models...

for some other tables, you have many normalization issues - for example:

  • you should have a separate address table (not part of the ad)
  • you should have a picture table (and link those to the ads with another table)
  • you should have a person table - and link that to the ad as 'owner'
  • the idea of 'favorite' should also be in this person->ad relationship table as a role or type column
Read More

Friday, March 11, 2016

How to know relations between tables

Leave a Comment

I have a database in MySQL created by someone. I don't have any documentation of the database.

How can I know the relationship between the tables?

Is there any query or a procedure to generate a report so that it's easy to find the relations?

I can look into Schema information and manually figure it out, but it would be great if I could generate a relationship report.

7 Answers

Answers 1

You can get an overview in MySql Workbench by doing the steps below:

  1. Go to "Database" Menu option.
  2. Select the "Reverse Engineer" option.
  3. A wizard will be opened and will generate an EER Diagram which shows up

Answers 2

The better way as programmatically speaking is gathering data from INFORMATION_SCHEMA.KEY_COLUMN_USAGE table as follows:

SELECT    `TABLE_SCHEMA`,                          -- Foreign key schema   `TABLE_NAME`,                            -- Foreign key table   `COLUMN_NAME`,                           -- Foreign key column   `REFERENCED_TABLE_SCHEMA`,               -- Origin key schema   `REFERENCED_TABLE_NAME`,                 -- Origin key table   `REFERENCED_COLUMN_NAME`                 -- Origin key column FROM   `INFORMATION_SCHEMA`.`KEY_COLUMN_USAGE`  -- Will fail if user don't have privilege WHERE   `TABLE_SCHEMA` = SCHEMA()                -- Detect current schema in USE    AND `REFERENCED_TABLE_NAME` IS NOT NULL; -- Only tables with foreign keys 

and another one is

select * from INFORMATION_SCHEMA.TABLE_CONSTRAINTS; 

Answers 3

Try out SchemaSpy (http://schemaspy.sourceforge.net/):

SchemaSpy is a Java-based tool (requires Java 5 or higher) that analyzes the metadata of a schema in a database and generates a visual representation of it in a browser-displayable format.

Here is a screenshot of the HTML page of the sample output from http://schemaspy.sourceforge.net/sample/ :

Screenshot of the HTML page of the sample output from http://schemaspy.sourceforge.net/sample/

There is also a nice GUI if you do not want to use the command line: http://schemaspygui.sourceforge.net/

Both tools are open source and in my opinion very lightweight and easy to use. I used them several times when I was in situations that you described: To get an overview of the schema and even some details to dive deeper. (Take a look at the "Anomalies" report.)

Answers 4

You may take a look at information_scheme.KEY_COLUMN_USAGE table

As it is suggested there a quick way to list your FKs (Foreign Key references) using the KEY_COLUMN_USAGE view:

SELECT CONCAT( table_name, '.', column_name, ' -> ', referenced_table_name, '.', referenced_column_name ) AS list_of_fks FROM information_schema.KEY_COLUMN_USAGE WHERE REFERENCED_TABLE_SCHEMA = (your schema name here) AND REFERENCED_TABLE_NAME is not null ORDER BY TABLE_NAME, COLUMN_NAME; 

Answers 5

Do you have the SELECTs that use the database? That may be the best source of the relationships.

Answers 6

If you are using phpmyadmin then:

  1. Goto the database.
  2. Select the table and goto its structure.
  3. You'll find relation view at the bottom of your table structure.

Answers 7

One more valuable option may be if you just install mysql workbench.( refers to) And try "Create EER models from database" .You will surely able to see relations among tables.

Read More