Streams/streams
Streams/streams
1
0
Fork 0
mirror of https://codeberg.org/streams/streams.git synced 2024-09-21 04:15:12 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
streams/include/dir_fns.php

298 lines
8.4 KiB
PHP
Raw Normal View History

start formatting for Doxygen
2013-02-26 01:09:40 +00:00
<?php /** @file */
directory page still needs a lot of work, look up locally if local master or standalone, need to put some basic profile info in discovery page and direct to local directory master
2012-12-20 03:51:52 +00:00
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
require_once('include/permissions.php');
directory page still needs a lot of work, look up locally if local master or standalone, need to put some basic profile info in discovery page and direct to local directory master
2012-12-20 03:51:52 +00:00
function find_upstream_directory($dirmode) {
Set a default directory server from a hard-wired list if one was not previously chosen.
2014-04-26 01:10:23 +00:00
global $DIRECTORY_FALLBACK_SERVERS;
provide config for preferred directory server
2013-10-14 02:29:54 +00:00
$preferred = get_config('system','directory_server');
Set a default directory server from a hard-wired list if one was not previously chosen.
2014-04-26 01:10:23 +00:00
if(! $preferred) {
/**
* No directory has yet been set. For most sites, pick one at random
* from our list of directory servers. However, if we're a directory
* server ourself, point at the local instance
* We will then set this value so this should only ever happen once.
* Ideally there will be an admin setting to change to a different
* directory server if you don't like our choice or if circumstances change.
*/
$dirmode = intval(get_config('system','directory_mode'));
if($dirmode == DIRECTORY_MODE_NORMAL) {
$toss = mt_rand(0,count($DIRECTORY_FALLBACK_SERVERS));
$preferred = $DIRECTORY_FALLBACK_SERVERS[$toss];
set_config('system','directory_server',$preferred);
}
else{
set_config('system','directory_server',z_root());
}
}
return array('url' => $preferred);
directory page still needs a lot of work, look up locally if local master or standalone, need to put some basic profile info in discovery page and direct to local directory master
2012-12-20 03:51:52 +00:00
}
Add check_upstream_directory() to check that your directory is still a directory, and if it isn't, let find_upstream_directory() find you a new one.
2014-08-31 15:58:17 +00:00
function check_upstream_directory() {
/**
* Directories may come and go over time. We will need to check that our
* directory server is still valid occasionally, and reset to something that
* is if our directory has gone offline for any reason
*/
$directory = get_config('system','directory_server');
if ($directory) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select * from site where site_url = '%s' and (site_flags & %d)>0 ",
Add check_upstream_directory() to check that your directory is still a directory, and if it isn't, let find_upstream_directory() find you a new one.
2014-08-31 15:58:17 +00:00
dbesc($directory),
intval(DIRECTORY_MODE_PRIMARY|DIRECTORY_MODE_SECONDARY|DIRECTORY_MODE_STANDALONE)
);
}
// If we've got something, it's still a directory. If we haven't, we need to reset and let find_upstream_directory() fix it
if (! $r) {
set_config('system','directory_server','');
}
return;
}
directory sort links on sidebar
2013-10-14 02:49:40 +00:00
function dir_sort_links() {
$o = replace_macros(get_markup_template('dir_sort_links.tpl'), array(
'$header' => t('Sort Options'),
'$normal' => t('Alphabetic'),
'$reverse' => t('Reverse Alphabetic'),
'$date' => t('Newest to Oldest')
));
return $o;
}
Fix missing argument warning
2013-11-12 21:26:47 +00:00
function dir_safe_mode() {
Bring safe search to the UI
2013-11-09 18:43:40 +00:00
$observer = get_observer_hash();
split private messages into two modules - "message" is just for message lists, "mail" is for reading and writing conversations. This is so we can Comanchify it cleanly.
2013-12-22 07:47:44 +00:00
if (! $observer)
return;
Clean up
2013-11-09 18:45:52 +00:00
if ($observer)
Bring safe search to the UI
2013-11-09 18:43:40 +00:00
$safe_mode = get_xconfig($observer,'directory','safe_mode');
More safe search - since the default is one, toggle only works after an initial xconfig is set - the first one will turn safe search on (ie, do nothing from a users point of view). Fix that.
2013-11-11 06:13:17 +00:00
if($safe_mode === '0')
Bring safe search to the UI
2013-11-09 18:43:40 +00:00
$toggle = t('Enable Safe Search');
else
$toggle = t('Disable Safe Search');
$o = replace_macros(get_markup_template('safesearch.tpl'), array(
turn safemode into a proper widget
2013-11-11 09:18:09 +00:00
'$safemode' => t('Safe Mode'),
Bring safe search to the UI
2013-11-09 18:43:40 +00:00
'$toggle' => $toggle,
));
return $o;
}
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
function sync_directories($dirmode) {
if($dirmode == DIRECTORY_MODE_STANDALONE || $dirmode == DIRECTORY_MODE_NORMAL)
return;
some more work on realms
2014-08-18 02:06:56 +00:00
$realm = get_directory_realm();
if($realm == DIRECTORY_REALM) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select * from site where (site_flags & %d)>0 and site_url != '%s' and ( site_realm = '%s' or site_realm = '') ",
some more work on realms
2014-08-18 02:06:56 +00:00
intval(DIRECTORY_MODE_PRIMARY|DIRECTORY_MODE_SECONDARY),
dbesc(z_root()),
dbesc($realm)
);
}
else {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select * from site where (site_flags & %d)>0 and site_url != '%s' and site_realm like '%s' ",
some more work on realms
2014-08-18 02:06:56 +00:00
intval(DIRECTORY_MODE_PRIMARY|DIRECTORY_MODE_SECONDARY),
dbesc(z_root()),
dbesc(protect_sprintf('%' . $realm . '%'))
);
}
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
// If there are no directory servers, setup the fallback master
some more work on realms
2014-08-18 02:06:56 +00:00
// FIXME - what to do if we're in a different realm?
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
if((! $r) && (z_root() != DIRECTORY_FALLBACK_MASTER)) {
$r = array(
'site_url' => DIRECTORY_FALLBACK_MASTER,
'site_flags' => DIRECTORY_MODE_PRIMARY,
This is long overdue - use a symblic constant NULL_DATE instead of the easily mis-typed sequence '0000-00-00 00:00:00'
2014-09-09 03:35:15 +00:00
'site_update' => NULL_DATE,
some more work on realms
2014-08-18 02:06:56 +00:00
'site_directory' => DIRECTORY_FALLBACK_MASTER . '/dirsearch',
'site_realm' => DIRECTORY_REALM
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
);
some more work on realms
2014-08-18 02:06:56 +00:00
$x = q("insert into site ( site_url, site_flags, site_update, site_directory, site_realm )
values ( '%s', %d', '%s', '%s', '%s' ) ",
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
dbesc($r[0]['site_url']),
intval($r[0]['site_flags']),
dbesc($r[0]['site_update']),
some more work on realms
2014-08-18 02:06:56 +00:00
dbesc($r[0]['site_directory']),
dbesc($r[0]['site_realm'])
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
);
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select * from site where (site_flags & %d)>0 and site_url != '%s'",
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
intval(DIRECTORY_MODE_PRIMARY|DIRECTORY_MODE_SECONDARY),
dbesc(z_root())
);
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
}
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if(! $r)
return;
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
foreach($r as $rr) {
if(! $rr['site_directory'])
continue;
limit updates from new dir servers to avoid exhausting memory.
2014-06-30 01:18:29 +00:00
log the offending dir server
2014-06-30 01:24:34 +00:00
logger('sync directories: ' . $rr['site_directory']);
limit updates from new dir servers to avoid exhausting memory.
2014-06-30 01:18:29 +00:00
// for brand new directory servers, only load the last couple of days. Everything before that will be repeats.
This is long overdue - use a symblic constant NULL_DATE instead of the easily mis-typed sequence '0000-00-00 00:00:00'
2014-09-09 03:35:15 +00:00
$syncdate = (($rr['site_sync'] === NULL_DATE) ? datetime_convert('UTC','UTC','now - 2 days') : $rr['site_sync']);
limit updates from new dir servers to avoid exhausting memory.
2014-06-30 01:18:29 +00:00
$x = z_fetch_url($rr['site_directory'] . '?f=&sync=' . urlencode($syncdate));
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if(! $x['success'])
continue;
$j = json_decode($x['body'],true);
if((! $j['transactions']) || (! is_array($j['transactions'])))
continue;
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update site set site_sync = '%s' where site_url = '%s'",
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
dbesc(datetime_convert()),
dbesc($rr['site_url'])
);
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
logger('sync_directories: ' . $rr['site_url'] . ': ' . print_r($j,true), LOGGER_DATA);
if(count($j['transactions'])) {
foreach($j['transactions'] as $t) {
$r = q("select * from updates where ud_guid = '%s' limit 1",
dbesc($t['transaction_id'])
);
if($r)
continue;
$ud_flags = 0;
if(is_array($t['flags']) && in_array('deleted',$t['flags']))
$ud_flags |= UPDATE_FLAGS_DELETED;
directory sync issues
2014-03-04 05:00:42 +00:00
if(is_array($t['flags']) && in_array('forced',$t['flags']))
$ud_flags |= UPDATE_FLAGS_FORCED;
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
$z = q("insert into updates ( ud_hash, ud_guid, ud_date, ud_flags, ud_addr )
sql typo in directory sync
2013-10-15 03:51:26 +00:00
values ( '%s', '%s', '%s', %d, '%s' ) ",
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
dbesc($t['hash']),
dbesc($t['transaction_id']),
dbesc($t['timestamp']),
intval($ud_flags),
dbesc($t['address'])
);
}
}
}
}
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
function update_directory_entry($ud) {
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
logger('update_directory_entry: ' . print_r($ud,true), LOGGER_DATA);
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if($ud['ud_addr'] && (! ($ud['ud_flags'] & UPDATE_FLAGS_DELETED))) {
extra logging
2013-10-15 04:56:56 +00:00
$success = false;
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
$x = zot_finger($ud['ud_addr'],'');
if($x['success']) {
$j = json_decode($x['body'],true);
extra logging
2013-10-15 04:56:56 +00:00
if($j)
$success = true;
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
$y = import_xchan($j,0,$ud);
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
}
extra logging
2013-10-15 04:56:56 +00:00
if(! $success) {
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
$r = q("update updates set ud_last = '%s' where ud_addr = '%s'",
dbesc(datetime_convert()),
dbesc($ud['ud_addr'])
);
}
}
create update record for accounts on directory server itself during profile changes, re-queue failed directory sync packets, fix account_id index in queue creation, be more selective about updating hubloc_connected (only when we're talking to the associated site). But we still need a way to flag dead accounts and these mechanisms won't cut it, because it isn't a requirement that channels communicate with anybody, except for contacting the directory when the account is created. I think we need to make every channel ping the directory once a month. Then we can find those that have not done so (after syncing with other directories). We can't have the directory just flag dead accounts as this would create a way for an anti-social attacker to become a directory server and mark all the accounts dead.
2013-10-25 02:38:50 +00:00
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
}
implement a forced directory update mode where we unconditionally create a directory sync packet. This is needed to ensure that monthly directory pings are propagated to other directory servers so they can each prove for themselves whether or not an account is alive or dead. We do not trust other directories to provide us information beyond "look at this entry and decide for yourself" as doing otherwise would invite rogue directory manipulations. As this scheduled update occurs on all channels across all servers, we should also pick up refresh messages from all existing channel clones and these should also propagate out to all directory servers using the same mechanism (though perhaps not at the same time).
2014-02-11 22:19:20 +00:00
/**
* @function local_dir_update($uid,$force)
* push local channel updates to a local directory server
*
*/
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
implement a forced directory update mode where we unconditionally create a directory sync packet. This is needed to ensure that monthly directory pings are propagated to other directory servers so they can each prove for themselves whether or not an account is alive or dead. We do not trust other directories to provide us information beyond "look at this entry and decide for yourself" as doing otherwise would invite rogue directory manipulations. As this scheduled update occurs on all channels across all servers, we should also pick up refresh messages from all existing channel clones and these should also propagate out to all directory servers using the same mechanism (though perhaps not at the same time).
2014-02-11 22:19:20 +00:00
function local_dir_update($uid,$force) {
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
implement a forced directory update mode where we unconditionally create a directory sync packet. This is needed to ensure that monthly directory pings are propagated to other directory servers so they can each prove for themselves whether or not an account is alive or dead. We do not trust other directories to provide us information beyond "look at this entry and decide for yourself" as doing otherwise would invite rogue directory manipulations. As this scheduled update occurs on all channels across all servers, we should also pick up refresh messages from all existing channel clones and these should also propagate out to all directory servers using the same mechanism (though perhaps not at the same time).
2014-02-11 22:19:20 +00:00
logger('local_dir_update', LOGGER_DEBUG);
directory debugging
2013-02-20 07:21:23 +00:00
create update record for accounts on directory server itself during profile changes, re-queue failed directory sync packets, fix account_id index in queue creation, be more selective about updating hubloc_connected (only when we're talking to the associated site). But we still need a way to flag dead accounts and these mechanisms won't cut it, because it isn't a requirement that channels communicate with anybody, except for contacting the directory when the account is created. I think we need to make every channel ping the directory once a month. Then we can find those that have not done so (after syncing with other directories). We can't have the directory just flag dead accounts as this would create a way for an anti-social attacker to become a directory server and mark all the accounts dead.
2013-10-25 02:38:50 +00:00
$p = q("select channel.channel_hash, channel_address, channel_timezone, profile.* from profile left join channel on channel_id = uid where uid = %d and is_default = 1",
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
intval($uid)
);
$profile = array();
add an encoding type to formatted zot communication structures so that we can later add alternate encodings.
2014-07-24 22:55:01 +00:00
$profile['encoding'] = 'zot';
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
if($p) {
$hash = $p[0]['channel_hash'];
$profile['description'] = $p[0]['pdesc'];
$profile['birthday'] = $p[0]['dob'];
really, truly fix the age missing in directory bug
2013-07-03 05:41:17 +00:00
if($age = age($p[0]['dob'],$p[0]['channel_timezone'],''))
$profile['age'] = $age;
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
$profile['gender'] = $p[0]['gender'];
$profile['marital'] = $p[0]['marital'];
$profile['sexual'] = $p[0]['sexual'];
$profile['locale'] = $p[0]['locality'];
$profile['region'] = $p[0]['region'];
$profile['postcode'] = $p[0]['postal_code'];
$profile['country'] = $p[0]['country_name'];
extend the directory profiles a bit more
2013-12-25 09:57:04 +00:00
$profile['about'] = $p[0]['about'];
$profile['homepage'] = $p[0]['homepage'];
$profile['hometown'] = $p[0]['hometown'];
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
if($p[0]['keywords']) {
$tags = array();
$k = explode(' ',$p[0]['keywords']);
if($k)
foreach($k as $kk)
if(trim($kk))
$tags[] = trim($kk);
if($tags)
$profile['keywords'] = $tags;
}
set xchan hidden flag correctly when local directory update
2013-02-20 07:31:26 +00:00
$hidden = (1 - intval($p[0]['publish']));
fix localdir privacy (again)
2013-02-20 07:41:42 +00:00
logger('hidden: ' . $hidden);
set xchan hidden flag correctly when local directory update
2013-02-20 07:31:26 +00:00
$r = q("select xchan_flags from xchan where xchan_hash = '%s' limit 1",
dbesc($p[0]['channel_hash'])
);
// Be careful - XCHAN_FLAGS_HIDDEN should evaluate to 1
if(($r[0]['xchan_flags'] & XCHAN_FLAGS_HIDDEN) != $hidden)
$new_flags = $r[0]['xchan_flags'] ^ XCHAN_FLAGS_HIDDEN;
else
$new_flags = $r[0]['xchan_flags'];
if($new_flags != $r[0]['xchan_flags']) {
fix localdir privacy (again)
2013-02-20 07:41:42 +00:00
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("update xchan set xchan_flags = %d where xchan_hash = '%s'",
set xchan hidden flag correctly when local directory update
2013-02-20 07:31:26 +00:00
intval($new_flags),
fix localdir privacy (again)
2013-02-20 07:41:42 +00:00
dbesc($p[0]['channel_hash'])
set xchan hidden flag correctly when local directory update
2013-02-20 07:31:26 +00:00
);
fix localdir privacy (again)
2013-02-20 07:41:42 +00:00
set xchan hidden flag correctly when local directory update
2013-02-20 07:31:26 +00:00
}
improve the directory popup a bit
2013-12-24 03:44:23 +00:00
$address = $p[0]['channel_address'] . '@' . get_app()->get_hostname();
set xchan hidden flag correctly when local directory update
2013-02-20 07:31:26 +00:00
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
if(perm_is_allowed($uid,'','view_profile')) {
improve the directory popup a bit
2013-12-24 03:44:23 +00:00
import_directory_profile($hash,$profile,$address,0);
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
}
else {
// they may have made it private
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("delete from xprof where xprof_hash = '%s'",
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
dbesc($hash)
);
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("delete from xtag where xtag_hash = '%s'",
This was bloody hard to sort out.... but now it's sorted and we can move forward again. Put back xtag for efficient keyword searching, use it in conjunction with xprof['keywords'] to avoid a SQL triple join and group_concat (which simply won't scale on a directory server), figured out how best to work timezone corrected birthdays into the protocol. The directory can even provide an age if one was provided to it. Both of these things need some more work, but we have the mechanisms and a plan how to do it right instead of "how the !@#$ are we going to do pull this off?"
2012-12-28 07:33:37 +00:00
dbesc($hash)
);
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
}
}
reversed args in update_modtime, incorrect ud_guid and ud_hash
2014-02-11 04:38:26 +00:00
$ud_hash = random_string() . '@' . get_app()->get_hostname();
regression: (-1) update_flag still exists in one function. This should be UPDATE_FLAGS_FORCED or 2.
2014-06-01 23:34:04 +00:00
update_modtime($hash,$ud_hash,$p[0]['channel_address'] . '@' . get_app()->get_hostname(),(($force) ? UPDATE_FLAGS_FORCED : UPDATE_FLAGS_UPDATED));
create update record for accounts on directory server itself during profile changes, re-queue failed directory sync packets, fix account_id index in queue creation, be more selective about updating hubloc_connected (only when we're talking to the associated site). But we still need a way to flag dead accounts and these mechanisms won't cut it, because it isn't a requirement that channels communicate with anybody, except for contacting the directory when the account is created. I think we need to make every channel ping the directory once a month. Then we can find those that have not done so (after syncing with other directories). We can't have the directory just flag dead accounts as this would create a way for an anti-social attacker to become a directory server and mark all the accounts dead.
2013-10-25 02:38:50 +00:00
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
}
Reference in a new issue Copy permalink