Streams/streams
Streams/streams
1
0
Fork 0
mirror of https://codeberg.org/streams/streams.git synced 2024-09-20 15:15:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
streams/include/poller.php

429 lines
11 KiB
PHP
Raw Normal View History

start formatting for Doxygen
2013-02-26 01:09:40 +00:00
<?php /** @file */
more work on email contacts
2011-04-16 06:40:43 +00:00
progress on poller
2012-12-31 00:14:29 +00:00
require_once('boot.php');
require_once('include/cli_startup.php');
modded scripts to be runned in cli so they can be included and executed in main program
2011-01-28 13:04:18 +00:00
allow polling to mostly survive minor memory shortages.
2011-07-01 04:56:07 +00:00
modded scripts to be runned in cli so they can be included and executed in main program
2011-01-28 13:04:18 +00:00
function poller_run($argv, $argc){
no more url in cmdline
2010-08-17 03:47:40 +00:00
progress on poller
2012-12-31 00:14:29 +00:00
cli_startup();
more work on email contacts
2011-04-16 06:40:43 +00:00
progress on poller
2012-12-31 00:14:29 +00:00
$a = get_app();
load db configs (config,system) for all "executables"
2011-06-30 08:15:18 +00:00
implement max load average before queuing/deferring delivery and poller processes
2012-05-04 04:50:48 +00:00
$maxsysload = intval(get_config('system','maxloadavg'));
if($maxsysload < 1)
$maxsysload = 50;
if(function_exists('sys_getloadavg')) {
$load = sys_getloadavg();
if(intval($load[0]) > $maxsysload) {
logger('system: load ' . $load . ' too high. Poller deferred to next scheduled run.');
return;
}
}
implement monthly directory pings
2013-10-28 07:11:44 +00:00
$interval = intval(get_config('system','poll_interval'));
if(! $interval)
$interval = ((get_config('system','delivery_interval') === false) ? 3 : intval(get_config('system','delivery_interval')));
Typo
2014-10-21 16:27:13 +00:00
// Check for a lockfile. If it exists, but is over an hour old, it's stale. Ignore it.
Prevent more than one instance of the poller running at a time.
2014-10-20 18:31:47 +00:00
$lockfile = 'store/[data]/poller';
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
if((file_exists($lockfile)) && (filemtime($lockfile) > (time() - 3600))
&& (! get_config('system','override_poll_lockfile'))) {
logger("poller: Already running");
Prevent more than one instance of the poller running at a time.
2014-10-20 18:31:47 +00:00
return;
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
}
Prevent more than one instance of the poller running at a time.
2014-10-20 18:31:47 +00:00
// Create a lockfile. Needs two vars, but $x doesn't need to contain anything.
file_put_contents($lockfile, $x);
implement monthly directory pings
2013-10-28 07:11:44 +00:00
log queue startup
2010-11-26 02:31:33 +00:00
logger('poller: start');
Mods to script runned via CLI to permits to run also included
2011-01-24 21:01:56 +00:00
change remote welcome message
2010-11-26 02:22:54 +00:00
// run queue delivery process in the background
schedule the queue re-delivery process
2010-11-22 23:30:52 +00:00
move php path setting inside proc_run
2011-02-23 23:16:12 +00:00
proc_run('php',"include/queue.php");
this query can take quite a while if there were many updates but it has not been called for a long time (eg nobody called mod sharedwithme). might be a good idea to have this in the poller to clean up on a regular basis and call it once more in mod sharedwithme. what do you think?
2015-03-24 09:56:02 +00:00
// maintenance for mod sharedwithme - check for updated items and remove them
require_once('include/sharedwithme.php');
apply_updates();
delete old cache in poller.php
2011-01-31 16:16:35 +00:00
allow private mail sender to set an expiration on their messages. Once expired the message is destroyed at both ends (subject to the granularity of the polling interval) and is gone. Officially it takes some form of language independent string like 2013/11/22, but English speakers can use anything that strtotime() understands, like "+30 minutes" or "next Tuesday".
2013-11-07 02:28:36 +00:00
// expire any expired mail
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("delete from mail where expires != '%s' and expires < %s ",
dbesc(NULL_DATE),
db_utcnow()
missed one
2014-09-09 03:45:17 +00:00
);
provide expire feature on post and comments as well as private messages. This requires a feature setting and is currently unmaskable - if the author says it's gone, there's not a lot you can do to stop it or try and save it. It's gone.
2013-11-07 08:47:50 +00:00
move expiration of notifications to the equivalent of "cron daily" to try and reduce the number of things the poller has to do on every run.
2014-02-11 22:35:02 +00:00
// expire any expired items
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select id from item where expires != '%s' and expires < %s
start on item_restrict conversion
2015-01-23 05:04:54 +00:00
and item_deleted = 0 ",
missed one
2014-09-09 03:45:17 +00:00
dbesc(NULL_DATE),
start on item_restrict conversion
2015-01-23 05:04:54 +00:00
db_utcnow()
we should probably only expire things once...
2013-11-08 06:45:13 +00:00
);
provide expire feature on post and comments as well as private messages. This requires a feature setting and is currently unmaskable - if the author says it's gone, there's not a lot you can do to stop it or try and save it. It's gone.
2013-11-07 08:47:50 +00:00
if($r) {
require_once('include/items.php');
foreach($r as $rr)
drop_item($rr['id'],false);
}
Clear out old notifications in the poller.
2014-02-11 20:14:45 +00:00
implement monthly directory pings
2013-10-28 07:11:44 +00:00
// Ensure that every channel pings a directory server once a month. This way we can discover
// channels and sites that quietly vanished and prevent the directory from accumulating stale
// or dead entries.
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select channel_id from channel where channel_dirdate < %s - INTERVAL %s",
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
db_utcnow(),
db_quoteinterval('30 DAY')
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
);
implement monthly directory pings
2013-10-28 07:11:44 +00:00
if($r) {
foreach($r as $rr) {
implement a forced directory update mode where we unconditionally create a directory sync packet. This is needed to ensure that monthly directory pings are propagated to other directory servers so they can each prove for themselves whether or not an account is alive or dead. We do not trust other directories to provide us information beyond "look at this entry and decide for yourself" as doing otherwise would invite rogue directory manipulations. As this scheduled update occurs on all channels across all servers, we should also pick up refresh messages from all existing channel clones and these should also propagate out to all directory servers using the same mechanism (though perhaps not at the same time).
2014-02-11 22:19:20 +00:00
proc_run('php','include/directory.php',$rr['channel_id'],'force');
implement monthly directory pings
2013-10-28 07:11:44 +00:00
if($interval)
@time_sleep_until(microtime(true) + (float) $interval);
}
}
implement time travelling posts
2013-05-24 00:24:15 +00:00
// publish any applicable items that were set to be published in the future
put a lower time limit on time travellers
2015-03-23 01:23:37 +00:00
// (time travel posts). Restrict to items that have come of age in the last
// couple of days to limit the query to something reasonable.
implement time travelling posts
2013-05-24 00:24:15 +00:00
more work on item table optimisation
2015-06-10 23:59:04 +00:00
$r = q("select id from item where item_delayed = 1 and created <= %s and created > '%s' ",
put a lower time limit on time travellers
2015-03-23 01:23:37 +00:00
db_utcnow(),
dbesc(datetime_convert('UTC','UTC','now - 2 days'))
implement time travelling posts
2013-05-24 00:24:15 +00:00
);
zidify img links, delay notifier until actually published for time travelling posts
2013-05-27 12:27:35 +00:00
if($r) {
foreach($r as $rr) {
more work on item table optimisation
2015-06-10 23:59:04 +00:00
$x = q("update item set item_delayed = 0 where id = %d",
zidify img links, delay notifier until actually published for time travelling posts
2013-05-27 12:27:35 +00:00
intval($rr['id'])
);
if($x) {
Typo
2014-02-27 08:54:07 +00:00
proc_run('php','include/notifier.php','wall-new',$rr['id']);
zidify img links, delay notifier until actually published for time travelling posts
2013-05-27 12:27:35 +00:00
}
}
}
implement time travelling posts
2013-05-24 00:24:15 +00:00
do not poll for abandoned accounts
2011-09-30 04:20:19 +00:00
$abandon_days = intval(get_config('system','account_abandon_days'));
if($abandon_days < 1)
$abandon_days = 0;
diaspora birthday notifications
2011-10-14 07:20:37 +00:00
// once daily run birthday_updates and then expire in background
content expiration
2011-03-16 00:31:49 +00:00
add age to directory profile - requires updating on each birthday and that part is still missing
2013-06-30 07:38:02 +00:00
// FIXME: add birthday updates, both locally and for xprof for use
// by directory servers
allow a site to run cron daily at a chosen hour
2014-07-16 23:21:01 +00:00
$d1 = intval(get_config('system','last_expire_day'));
content expiration
2011-03-16 00:31:49 +00:00
$d2 = intval(datetime_convert('UTC','UTC','now','d'));
allow a site to run cron daily at a chosen hour
2014-07-16 23:21:01 +00:00
// Allow somebody to staggger daily activities if they have more than one site on their server,
// or if it happens at an inconvenient (busy) hour.
$h1 = intval(get_config('system','cron_hour'));
$h2 = intval(datetime_convert('UTC','UTC','now','G'));
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
$dirmode = get_config('system','directory_mode');
enable network/matrix expiration, this should be functional but the options have been reduced/restricted so we're only looking at network posts and ignore anything that is filed, starred, or is resource_type 'photo' (which should not be possible in non-wall posts, but we just want to be sure). Will require the adventurous tester(s) to set 'channel_expire_days' in their channel record.
2014-03-04 03:15:11 +00:00
/**
* Cron Daily
*
* Actions in the following block are executed once per day, not on every poller run
*
*/
move expiration of notifications to the equivalent of "cron daily" to try and reduce the number of things the poller has to do on every run.
2014-02-11 22:35:02 +00:00
allow a site to run cron daily at a chosen hour
2014-07-16 23:21:01 +00:00
if(($d2 != $d1) && ($h1 == $h2)) {
diaspora birthday notifications
2011-10-14 07:20:37 +00:00
Provide a way to mark photos as adult and hide them from the default album view. Still need a button or setting to enable "unsafe viewing". This has no effect anywhere but in the album views. They can still be viewed by flipping through the individual photos with 'prev' and 'next'. We probably need a comprehensive strategy for how to deal with n-s-f-w photos in albums so consider this a band-aid which requires additional work and integration with other facilities which access these photos. It is entirely optional.
2014-11-07 01:27:28 +00:00
require_once('include/dir_fns.php');
check_upstream_directory();
Add check_upstream_directory() to check that your directory is still a directory, and if it isn't, let find_upstream_directory() find you a new one.
2014-08-31 15:58:17 +00:00
doc updates
2014-06-07 06:56:52 +00:00
call_hooks('cron_daily',datetime_convert());
prune_hub_reinstalls() and add cron weekly as a side effect
2014-03-20 02:53:14 +00:00
$d3 = intval(datetime_convert('UTC','UTC','now','N'));
if($d3 == 7) {
/**
* Cron Weekly
*
* Actions in the following block are executed once per day only on Sunday (once per week).
*
*/
doc updates
2014-06-07 06:56:52 +00:00
call_hooks('cron_weekly',datetime_convert());
Nag hub admins with self-signed certs to fix them. Otherwise we'll just have to start marking them as dead sites.
2015-03-02 04:14:42 +00:00
z_check_cert();
doc updates
2014-06-07 06:56:52 +00:00
prune_hub_reinstalls() and add cron weekly as a side effect
2014-03-20 02:53:14 +00:00
require_once('include/hubloc.php');
prune_hub_reinstalls();
cleanup dead directory entries. This was a real b#tch, so keep your eye out for issues - which you shouldn't see until next weekend when this is scheduled to run. We're only setting flags, so if anything goes wrong we should be able to recover without too much pain.
2014-05-29 04:42:46 +00:00
require_once('include/Contact.php');
mark_orphan_hubsxchans();
prune_hub_reinstalls() and add cron weekly as a side effect
2014-03-20 02:53:14 +00:00
allow a site to run cron daily at a chosen hour
2014-07-16 23:21:01 +00:00
get rid of really old poco records once weekly
2015-01-20 22:04:44 +00:00
// get rid of really old poco records
provide storage for directory based reputation in the xlink table by setting xlink_static = 1, so that xlink_static = 0 is traditional poco linkages
2015-01-27 03:13:06 +00:00
q("delete from xlink where xlink_updated < %s - INTERVAL %s and xlink_static = 0 ",
get rid of really old poco records once weekly
2015-01-20 22:04:44 +00:00
db_utcnow(), db_quoteinterval('14 DAY')
);
two tracks proceeding in parallel - first adding the ability to store a digital signature with ratings so that directories can pass them around, second provide some directory registration functions so we can obtain a list of directories from somewhere (to pass ratings around between them). This gives the primary directory role some value as that is where you register your directory.
2015-02-01 23:10:52 +00:00
$dirmode = intval(get_config('system','directory_mode'));
fix regdir
2015-04-01 04:06:48 +00:00
if($dirmode === DIRECTORY_MODE_SECONDARY || $dirmode === DIRECTORY_MODE_PRIMARY) {
tweaks to directory registrations which don't seem to be happening, also downgrade any directory that claims to be a primary directory for RED_GLOBAL.
2015-03-19 22:56:30 +00:00
logger('regdir: ' . print_r(z_fetch_url(get_directory_primary() . '/regdir?f=&url=' . urlencode(z_root()) . '&realm=' . urlencode(get_directory_realm())),true));
two tracks proceeding in parallel - first adding the ability to store a digital signature with ratings so that directories can pass them around, second provide some directory registration functions so we can obtain a list of directories from somewhere (to pass ratings around between them). This gives the primary directory role some value as that is where you register your directory.
2015-02-01 23:10:52 +00:00
}
get rid of really old poco records once weekly
2015-01-20 22:04:44 +00:00
allow a site to run cron daily at a chosen hour
2014-07-16 23:21:01 +00:00
/**
* End Cron Weekly
*/
prune_hub_reinstalls() and add cron weekly as a side effect
2014-03-20 02:53:14 +00:00
}
bring back birthdays
2014-06-03 00:49:19 +00:00
update_birthdays();
query for statistics are now executed by poller
2014-10-02 11:43:48 +00:00
//update statistics in config
require_once('include/statistics_fns.php');
update_channels_total_stat();
update_channels_active_halfyear_stat();
update_channels_active_monthly_stat();
update_local_posts_stat();
move expiration of notifications to the equivalent of "cron daily" to try and reduce the number of things the poller has to do on every run.
2014-02-11 22:35:02 +00:00
// expire any read notifications over a month old
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("delete from notify where seen = 1 and date < %s - INTERVAL %s",
db_utcnow(), db_quoteinterval('30 DAY')
);
move expiration of notifications to the equivalent of "cron daily" to try and reduce the number of things the poller has to do on every run.
2014-02-11 22:35:02 +00:00
service class downgrade to the default service class on account expiration if using a non-default service class and account has expired.
2014-02-18 23:17:18 +00:00
// expire any expired accounts
downgrade_accounts();
move expiration of notifications to the equivalent of "cron daily" to try and reduce the number of things the poller has to do on every run.
2014-02-11 22:35:02 +00:00
for directory sync, ensure we have a fallback master; even if we have nothing else
2013-04-19 03:47:35 +00:00
// If this is a directory server, request a sync with an upstream
// directory at least once a day, up to once every poll interval.
// Pull remote changes and push local changes.
// potential issue: how do we keep from creating an endless update loop?
if($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
require_once('include/dir_fns.php');
sync_directories($dirmode);
}
content expiration
2011-03-16 00:31:49 +00:00
set_config('system','last_expire_day',$d2);
populate some posts when somebody is granted "read stream" permission
2013-11-18 00:50:32 +00:00
enable network/matrix expiration, this should be functional but the options have been reduced/restricted so we're only looking at network posts and ignore anything that is filed, starred, or is resource_type 'photo' (which should not be possible in non-wall posts, but we just want to be sure). Will require the adventurous tester(s) to set 'channel_expire_days' in their channel record.
2014-03-04 03:15:11 +00:00
proc_run('php','include/expire.php');
populate initial suggestions (ultimately we want to do this at install time as well as from the poller so that new sites have friend suggestions when they create their first channel).
2013-08-06 03:08:35 +00:00
proc_run('php','include/cli_suggest.php');
provide a way to sync locations and get rid of bogus hublocs, now implemented
2014-09-15 05:19:19 +00:00
require_once('include/hubloc.php');
remove_obsolete_hublocs();
allow a site to run cron daily at a chosen hour
2014-07-16 23:21:01 +00:00
/**
* End Cron Daily
*/
content expiration
2011-03-16 00:31:49 +00:00
}
fix buggered contact photos
2013-06-04 05:34:35 +00:00
// update any photos which didn't get imported properly
// This should be rare
$r = q("select xchan_photo_l, xchan_hash from xchan where xchan_photo_l != '' and xchan_photo_m = ''
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
and xchan_photo_date < %s - INTERVAL %s",
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
db_utcnow(),
db_quoteinterval('1 DAY')
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
);
fix buggered contact photos
2013-06-04 05:34:35 +00:00
if($r) {
require_once('include/photo/photo_driver.php');
foreach($r as $rr) {
$photos = import_profile_photo($rr['xchan_photo_l'],$rr['xchan_hash']);
$x = q("update xchan set xchan_photo_l = '%s', xchan_photo_m = '%s', xchan_photo_s = '%s', xchan_photo_mimetype = '%s'
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
where xchan_hash = '%s'",
fix buggered contact photos
2013-06-04 05:34:35 +00:00
dbesc($photos[0]),
dbesc($photos[1]),
dbesc($photos[2]),
dbesc($photos[3]),
dbesc($rr['xchan_hash'])
);
}
}
more work on firehose
2014-03-27 01:45:01 +00:00
firehose fixes and optimisations. In particular get rid of the unresponsive script warning when trying to load updates (sine they aren't our posts, we can't check for unseen, hence we can't really load updates). Also make the url selection pluggable.
2014-03-27 22:35:29 +00:00
// pull in some public posts
allow a site to run cron daily at a chosen hour
2014-07-16 23:21:01 +00:00
if(! get_config('system','disable_discover_tab'))
Let the site admin choose whether to display the Discover tab.
2014-03-30 02:20:43 +00:00
proc_run('php','include/externals.php');
more work on firehose
2014-03-27 01:45:01 +00:00
Cache: item cache now has an autodelete of old files. The pages plugin now sets a link via redirection. So posting works.
2012-03-11 18:11:25 +00:00
silence php notices about $manual_id not being defined
2011-02-03 03:56:27 +00:00
$manual_id = 0;
allow polling to mostly survive minor memory shortages.
2011-07-01 04:56:07 +00:00
$generation = 0;
more DB cleanup
2012-08-31 01:47:07 +00:00
silence php notices about $manual_id not being defined
2011-02-03 03:56:27 +00:00
$force = false;
allow polling to mostly survive minor memory shortages.
2011-07-01 04:56:07 +00:00
$restart = false;
add page start/end hooks, cleanup couple of notice msgs
2011-01-10 21:45:42 +00:00
issues marking for death
2013-08-16 00:36:00 +00:00
if(($argc > 1) && ($argv[1] == 'force'))
hub anomolies & swat0 compliance
2010-11-05 03:47:44 +00:00
$force = true;
issues marking for death
2013-08-16 00:36:00 +00:00
if(($argc > 1) && ($argv[1] == 'restart')) {
allow polling to mostly survive minor memory shortages.
2011-07-01 04:56:07 +00:00
$restart = true;
issues marking for death
2013-08-16 00:36:00 +00:00
$generation = intval($argv[2]);
allow polling to mostly survive minor memory shortages.
2011-07-01 04:56:07 +00:00
if(! $generation)
killme();
}
issues marking for death
2013-08-16 00:36:00 +00:00
if(($argc > 1) && intval($argv[1])) {
$manual_id = intval($argv[1]);
silence php notices about $manual_id not being defined
2011-02-03 03:56:27 +00:00
$force = true;
following random feeds
2011-01-24 04:09:34 +00:00
}
break up poller into separate processes
2012-04-28 06:17:40 +00:00
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
$sql_extra = (($manual_id) ? " AND abook_id = " . intval($manual_id) . " " : "");
following random feeds
2011-01-24 04:09:34 +00:00
lots of facebook fixes
2011-04-26 11:39:27 +00:00
reload_plugins();
reload plugins if they change
2011-03-09 10:12:32 +00:00
$d = datetime_convert();
lots of facebook fixes
2011-04-26 11:39:27 +00:00
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
// TODO check to see if there are any cronhooks before wasting a process
progress on poller
2012-12-31 00:14:29 +00:00
allow polling to mostly survive minor memory shortages.
2011-07-01 04:56:07 +00:00
if(! $restart)
split off facebook and other cron hooks from poller
2011-08-15 05:59:34 +00:00
proc_run('php','include/cronhooks.php');
reload plugins if they change
2011-03-09 10:12:32 +00:00
change flags for one-way connections from hidden to unconnected so we can still use hidden for - well hiding connections
2014-03-04 22:35:42 +00:00
// Only poll from those with suitable relationships
diaspora photos ?
2011-08-24 01:17:35 +00:00
do not poll for abandoned accounts
2011-09-30 04:20:19 +00:00
$abandon_sql = (($abandon_days)
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
? sprintf(" AND account_lastlog > %s - INTERVAL %s ", db_utcnow(), db_quoteinterval(intval($abandon_days).' DAY'))
do not poll for abandoned accounts
2011-09-30 04:20:19 +00:00
: ''
);
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
$randfunc = db_getfunc('RAND');
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
convert the abook fields
2015-06-15 04:08:00 +00:00
$contacts = q("SELECT * FROM abook LEFT JOIN xchan on abook_xchan = xchan_hash
LEFT JOIN account on abook_account = account_id
where abook_self = 0
following random feeds
2011-01-24 04:09:34 +00:00
$sql_extra
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
AND (( account_flags = %d ) OR ( account_flags = %d )) $abandon_sql ORDER BY $randfunc",
refresh now works on poller runs
2012-12-31 07:51:39 +00:00
intval(ACCOUNT_OK),
intval(ACCOUNT_UNVERIFIED) // FIXME
progress on poller
2012-12-31 00:14:29 +00:00
icon changes, feed security improvements
2011-03-05 04:55:32 +00:00
);
poller
2010-07-19 03:49:54 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if($contacts) {
poller
2010-07-19 03:49:54 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
foreach($contacts as $contact) {
progress on poller
2012-12-31 00:14:29 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
$update = false;
progress on poller
2012-12-31 00:14:29 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
$t = $contact['abook_updated'];
$c = $contact['abook_connected'];
progress on poller
2012-12-31 00:14:29 +00:00
convert the abook fields
2015-06-15 04:08:00 +00:00
if(intval($contact['abook_feed'])) {
implement service class for feed polling frequenecy, fixed a service class bug, moved service class stuff from plugin to account.php where it belongs and load that by default instead of on demand
2014-09-04 02:07:14 +00:00
$min = service_class_fetch($contact['abook_channel'],'minimum_feedcheck_minutes');
if(! $min)
$min = intval(get_config('system','minimum_feedcheck_minutes'));
get out the sunnies
2014-09-01 05:15:00 +00:00
if(! $min)
$min = 60;
$x = datetime_convert('UTC','UTC',"now - $min minutes");
if($c < $x) {
proc_run('php','include/onepoll.php',$contact['abook_id']);
if($interval)
@time_sleep_until(microtime(true) + (float) $interval);
}
continue;
}
mark for death implemented
2013-01-30 03:28:19 +00:00
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
if($contact['xchan_network'] !== 'zot')
continue;
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if($c == $t) {
if(datetime_convert('UTC','UTC', 'now') > datetime_convert('UTC','UTC', $t . " + 1 day"))
$update = true;
mark for death implemented
2013-01-30 03:28:19 +00:00
}
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
else {
Provide a way to mark photos as adult and hide them from the default album view. Still need a button or setting to enable "unsafe viewing". This has no effect anywhere but in the album views. They can still be viewed by flipping through the individual photos with 'prev' and 'next'. We probably need a comprehensive strategy for how to deal with n-s-f-w photos in albums so consider this a band-aid which requires additional work and integration with other facilities which access these photos. It is entirely optional.
2014-11-07 01:27:28 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
// if we've never connected with them, start the mark for death countdown from now
This is long overdue - use a symblic constant NULL_DATE instead of the easily mis-typed sequence '0000-00-00 00:00:00'
2014-09-09 03:35:15 +00:00
if($c == NULL_DATE) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("update abook set abook_connected = '%s' where abook_id = %d",
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
dbesc(datetime_convert()),
intval($contact['abook_id'])
);
$c = datetime_convert();
$update = true;
}
// He's dead, Jim
if(strcmp(datetime_convert('UTC','UTC', 'now'),datetime_convert('UTC','UTC', $c . " + 30 day")) > 0) {
convert the abook fields
2015-06-15 04:08:00 +00:00
$r = q("update abook set abook_archived = 1 where abook_id = %d",
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
intval($contact['abook_id'])
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
);
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
$update = false;
continue;
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
}
convert the abook fields
2015-06-15 04:08:00 +00:00
if(intval($contact['abook_archived'])) {
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
$update = false;
continue;
}
// might be dead, so maybe don't poll quite so often
// recently deceased, so keep up the regular schedule for 3 days
if((strcmp(datetime_convert('UTC','UTC', 'now'),datetime_convert('UTC','UTC', $c . " + 3 day")) > 0)
&& (strcmp(datetime_convert('UTC','UTC', 'now'),datetime_convert('UTC','UTC', $t . " + 1 day")) > 0))
$update = true;
mark for death implemented
2013-01-30 03:28:19 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
// After that back off and put them on a morphine drip
if(strcmp(datetime_convert('UTC','UTC', 'now'),datetime_convert('UTC','UTC', $t . " + 2 day")) > 0) {
$update = true;
}
mark for death implemented
2013-01-30 03:28:19 +00:00
No point beating a dead horse
2013-09-13 04:54:15 +00:00
}
convert the abook fields
2015-06-15 04:08:00 +00:00
if(intval($contact['abook_pending']) || intval($contact['abook_archived']) || intval($contact['abook_ignored']) || intval($contact['abook_blocked']))
some poller optimisations and a fix for undefined (empty) dbtype which shows up as a mysql error that 'rand' isn't found.
2014-11-16 23:51:42 +00:00
continue;
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if((! $update) && (! $force))
continue;
mark for death implemented
2013-01-30 03:28:19 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
proc_run('php','include/onepoll.php',$contact['abook_id']);
if($interval)
@time_sleep_until(microtime(true) + (float) $interval);
mark for death implemented
2013-01-30 03:28:19 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
}
}
mark for death implemented
2013-01-30 03:28:19 +00:00
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
fix dir server query take 2 rationale: use latest entry rather than latest update. update timestamp is updated for all entries upon success so they are equivalent. prevents dupes from fake null dates.
2015-01-22 17:25:34 +00:00
$r = q("SELECT u.ud_addr, u.ud_id, u.ud_last FROM updates AS u INNER JOIN (SELECT ud_addr, max(ud_id) AS ud_id FROM updates WHERE ( ud_flags & %d ) = 0 AND ud_addr != '' AND ( ud_last = '%s' OR ud_last > %s - INTERVAL %s ) GROUP BY ud_addr) AS s ON s.ud_id = u.ud_id ",
one more time
2014-09-09 03:47:36 +00:00
intval(UPDATE_FLAGS_UPDATED),
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
dbesc(NULL_DATE),
db_utcnow(), db_quoteinterval('7 DAY')
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
);
if($r) {
foreach($r as $rr) {
// If they didn't respond when we attempted before, back off to once a day
// After 7 days we won't bother anymore
This is long overdue - use a symblic constant NULL_DATE instead of the easily mis-typed sequence '0000-00-00 00:00:00'
2014-09-09 03:35:15 +00:00
if($rr['ud_last'] != NULL_DATE)
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if($rr['ud_last'] > datetime_convert('UTC','UTC', 'now - 1 day'))
continue;
proc_run('php','include/onedirsync.php',$rr['ud_id']);
if($interval)
@time_sleep_until(microtime(true) + (float) $interval);
mark for death implemented
2013-01-30 03:28:19 +00:00
}
}
icon changes, feed security improvements
2011-03-05 04:55:32 +00:00
}
Prevent more than one instance of the poller running at a time.
2014-10-20 18:31:47 +00:00
provide last successful poll run time (using relative time) in siteinfo so that we can tell folks that have no idea what we're talking about that their poller definitely isn't running and this is probably why nothing works correctly.
2014-12-22 02:40:25 +00:00
set_config('system','lastpoll',datetime_convert());
Prevent more than one instance of the poller running at a time.
2014-10-20 18:31:47 +00:00
//All done - clear the lockfile
@unlink($lockfile);
modded scripts to be runned in cli so they can be included and executed in main program
2011-01-28 13:04:18 +00:00
return;
}
poller
2010-07-19 03:49:54 +00:00
modded scripts to be runned in cli so they can be included and executed in main program
2011-01-28 13:04:18 +00:00
if (array_search(__file__,get_included_files())===0){
poller_run($argv,$argc);
killme();
}
Reference in a new issue Copy permalink