Streams/streams
Streams/streams
1
0
Fork 0
mirror of https://codeberg.org/streams/streams.git synced 2024-09-20 18:35:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
streams/include/zot.php

2972 lines
94 KiB
PHP
Raw Normal View History

control chars crept in.
2014-06-30 10:52:20 +00:00
<?php /** @file */
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
got zot
2012-11-13 04:59:18 +00:00
require_once('include/crypto.php');
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
require_once('include/items.php');
new function hubloc_change_primary()
2014-10-13 22:27:03 +00:00
require_once('include/hubloc.php');
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
item store hooks
2013-11-29 02:10:04 +00:00
/**
php warnings
2014-11-03 03:22:18 +00:00
* Red implementation of zot protocol.
item store hooks
2013-11-29 02:10:04 +00:00
*
* https://github.com/friendica/red/wiki/zot
* https://github.com/friendica/red/wiki/Zot---A-High-Level-Overview
*
*/
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
/**
*
moving a lot of structure around. 'entity' is now 'channel'
2012-09-26 00:57:20 +00:00
* @function zot_new_uid($channel_nick)
doco
2013-11-29 01:47:33 +00:00
*
* Generates a unique string for use as a zot guid using our DNS-based url, the channel nickname and some entropy.
* The entropy ensures uniqueness against re-installs where the same URL and nickname are chosen.
* NOTE: zot doesn't require this to be unique. Internally we use a whirlpool hash of this guid and the signature
php warnings
2014-11-03 03:22:18 +00:00
* of this guid signed with the channel private key. This can be verified and should make the probability of
* collision of the verified result negligible within the constraints of our immediate universe.
doco
2013-11-29 01:47:33 +00:00
*
* @param string channel_nickname = unique nickname of controlling entity
*
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
* @returns string
*
*/
moving a lot of structure around. 'entity' is now 'channel'
2012-09-26 00:57:20 +00:00
function zot_new_uid($channel_nick) {
$rawstr = z_root() . '/' . $channel_nick . '.' . mt_rand();
a few minor changes
2012-08-27 06:05:00 +00:00
return(base64url_encode(hash('whirlpool',$rawstr,true),true));
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
}
clean up more code duplication
2014-07-15 04:21:24 +00:00
/**
*
* function make_xchan_hash($guid,$guid_sig)
*
* Generates a portable hash identifier for the channel identified by $guid and signed with $guid_sig
* This ID is portable across the network but MUST be calculated locally by verifying the signature
* and can not be trusted as an identity.
*
*/
function make_xchan_hash($guid,$guid_sig) {
return base64url_encode(hash('whirlpool',$guid . $guid_sig, true));
}
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
/**
reduce the likelihood that a given channel will have 30-40 valid hublocs with the same hubloc_url.
2013-11-27 10:51:16 +00:00
* @function zot_get_hublocs($hash)
php warnings
2014-11-03 03:22:18 +00:00
* Given a zot hash, return all distinct hubs.
only list undeleted hublocs in zot_finger
2013-11-29 00:46:47 +00:00
* This function is used in building the zot discovery packet
doco
2013-11-29 01:47:33 +00:00
* and therefore should only be used by channels which are defined
* on this hub
reduce the likelihood that a given channel will have 30-40 valid hublocs with the same hubloc_url.
2013-11-27 10:51:16 +00:00
* @param string $hash - xchan_hash
doco
2013-11-29 01:47:33 +00:00
* @retuns array of hubloc (hub location structures)
* hubloc_id int
* hubloc_guid char(255)
* hubloc_guid_sig text
* hubloc_hash char(255)
* hubloc_addr char(255)
* hubloc_flags int
* hubloc_status int
* hubloc_url char(255)
* hubloc_url_sig text
* hubloc_host char(255)
* hubloc_callback char(255)
* hubloc_connect char(255)
* hubloc_sitekey text
* hubloc_updated datetime
php warnings
2014-11-03 03:22:18 +00:00
* hubloc_connected datetime
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
*
*/
reduce the likelihood that a given channel will have 30-40 valid hublocs with the same hubloc_url.
2013-11-27 10:51:16 +00:00
function zot_get_hublocs($hash) {
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
only list undeleted hublocs in zot_finger
2013-11-29 00:46:47 +00:00
/** Only search for active hublocs - e.g. those that haven't been marked deleted */
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$ret = q("select * from hubloc where hubloc_hash = '%s' and not ( hubloc_flags & %d )>0 order by hubloc_url ",
only list undeleted hublocs in zot_finger
2013-11-29 00:46:47 +00:00
dbesc($hash),
intval(HUBLOC_FLAGS_DELETED)
);
reduce the likelihood that a given channel will have 30-40 valid hublocs with the same hubloc_url.
2013-11-27 10:51:16 +00:00
return $ret;
move today's events into a ping notification
2012-07-29 00:21:42 +00:00
}
php warnings
2014-11-03 03:22:18 +00:00
doco
2013-11-29 01:47:33 +00:00
/**
*
* @function zot_build_packet($channel,$type = 'notify',$recipients = null, $remote_key = null, $secret = null)
* builds a zot notification packet that you can either
php warnings
2014-11-03 03:22:18 +00:00
* store in the queue with a message array or call zot_zot to immediately
doco
2013-11-29 01:47:33 +00:00
* zot it to the other side
more infrastructure changes
2012-11-15 03:27:16 +00:00
*
doco
2013-11-29 01:47:33 +00:00
* @param array $channel => sender channel structure
implement a forced directory update mode where we unconditionally create a directory sync packet. This is needed to ensure that monthly directory pings are propagated to other directory servers so they can each prove for themselves whether or not an account is alive or dead. We do not trust other directories to provide us information beyond "look at this entry and decide for yourself" as doing otherwise would invite rogue directory manipulations. As this scheduled update occurs on all channels across all servers, we should also pick up refresh messages from all existing channel clones and these should also propagate out to all directory servers using the same mechanism (though perhaps not at the same time).
2014-02-11 22:19:20 +00:00
* @param string $type => packet type: one of 'ping', 'pickup', 'purge', 'refresh', 'force_refresh', 'notify', 'auth_check'
doco
2013-11-29 02:46:59 +00:00
* @param array $recipients => envelope information, array ( 'guid' => string, 'guid_sig' => string ); empty for public posts
doco
2013-11-29 01:47:33 +00:00
* @param string $remote_key => optional public site key of target hub used to encrypt entire packet
php warnings
2014-11-03 03:22:18 +00:00
* NOTE: remote_key and encrypted packets are required for 'auth_check' packets, optional for all others
doco
2013-11-29 01:47:33 +00:00
* @param string $secret => random string, required for packets which require verification/callback
php warnings
2014-11-03 03:22:18 +00:00
* e.g. 'pickup', 'purge', 'notify', 'auth_check'. Packet types 'ping', 'force_refresh', and 'refresh' do not require verification
more infrastructure changes
2012-11-15 03:27:16 +00:00
*
doco
2013-11-29 01:47:33 +00:00
* @returns string json encoded zot packet
more infrastructure changes
2012-11-15 03:27:16 +00:00
*/
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
function zot_build_packet($channel,$type = 'notify',$recipients = null, $remote_key = null, $secret = null, $extra = null) {
more infrastructure changes
2012-11-15 03:27:16 +00:00
$data = array(
'type' => $type,
'sender' => array(
'guid' => $channel['channel_guid'],
'guid_sig' => base64url_encode(rsa_sign($channel['channel_guid'],$channel['channel_prvkey'])),
'url' => z_root(),
'url_sig' => base64url_encode(rsa_sign(z_root(),$channel['channel_prvkey']))
),
'callback' => '/post',
'version' => ZOT_REVISION
);
works one way sort of - now to go the other way - which will require fixing a whole slew of FIXMEs
2014-08-27 01:58:48 +00:00
if($recipients) {
for($x = 0; $x < count($recipients); $x ++)
unset($recipients[$x]['hash']);
more infrastructure changes
2012-11-15 03:27:16 +00:00
$data['recipients'] = $recipients;
works one way sort of - now to go the other way - which will require fixing a whole slew of FIXMEs
2014-08-27 01:58:48 +00:00
}
more infrastructure changes
2012-11-15 03:27:16 +00:00
magic auth - it's mostly done or at least all the code bits are written and it looks in theory to be pretty secure and it doesn't white screen. Getting it to actually work(?), well we won't know how hard that will be until we get it on a couple of systems and try it. Magic auth on one box is a no-op because you're already authenticated.
2013-01-22 03:16:21 +00:00
if($secret) {
For private messages, use hush-hush ultra top-secret mode by default
2012-11-19 03:19:16 +00:00
$data['secret'] = $secret;
magic auth - it's mostly done or at least all the code bits are written and it looks in theory to be pretty secure and it doesn't white screen. Getting it to actually work(?), well we won't know how hard that will be until we get it on a couple of systems and try it. Magic auth on one box is a no-op because you're already authenticated.
2013-01-22 03:16:21 +00:00
$data['secret_sig'] = base64url_encode(rsa_sign($secret,$channel['channel_prvkey']));
}
For private messages, use hush-hush ultra top-secret mode by default
2012-11-19 03:19:16 +00:00
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
if($extra) {
foreach($extra as $k => $v)
$data[$k] = $v;
}
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('zot_build_packet: ' . print_r($data,true), LOGGER_DATA);
tracking through comments
2012-11-27 11:08:26 +00:00
more infrastructure changes
2012-11-15 03:27:16 +00:00
// Hush-hush ultra top-secret mode
if($remote_key) {
Protocol: now set data['alg'] on all encapsulated encrypted packets, so that we can more easily retire 'aes256cbc' once it is no longer viable.
2013-11-20 23:20:12 +00:00
$data = crypto_encapsulate(json_encode($data),$remote_key);
more infrastructure changes
2012-11-15 03:27:16 +00:00
}
return json_encode($data);
}
doco
2013-03-26 04:32:12 +00:00
/**
* @function: zot_zot
* @param: string $url
* @param: array $data
*
* @returns: array => see z_post_url for returned data format
*/
more infrastructure changes
2012-11-15 03:27:16 +00:00
function zot_zot($url,$data) {
return z_post_url($url,array('data' => $data));
}
doco
2013-03-26 04:32:12 +00:00
/**
* @function: zot_finger
*
* Look up information about channel
* @param: string $webbie
* does not have to be host qualified e.g. 'foo' is treated as 'foo@thishub'
* @param: array $channel
* (optional), if supplied permissions will be enumerated specifically for $channel
bring back photo shadows (via theme settings)
2013-10-03 05:59:58 +00:00
* @param: boolean $autofallback
* fallback/failover to http if https connection cannot be established. Default is true.
doco
2013-03-26 04:32:12 +00:00
*
* @returns: array => see z_post_url and mod/zfinger.php
*/
in probe diagnostic - note https failure as diagnostic info - this also changes zot_finger so you can tell it not to fallback. This could be useful in corporate environments that are off the grid or other high security situations (as much as anything could be considered high security which relies on SSL).
2013-10-02 05:20:09 +00:00
function zot_finger($webbie,$channel,$autofallback = true) {
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
network discovery on follow
2012-11-01 04:42:20 +00:00
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
if(strpos($webbie,'@') === false) {
$address = $webbie;
$host = get_app()->get_hostname();
}
else {
$address = substr($webbie,0,strpos($webbie,'@'));
$host = substr($webbie,strpos($webbie,'@')+1);
}
$xchan_addr = $address . '@' . $host;
make zot_finger more bulletproof
2013-07-22 05:39:21 +00:00
if((! $address) || (! $xchan_addr)) {
logger('zot_finger: no address :' . $webbie);
return array('success' => false);
}
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
logger('using xchan_addr: ' . $xchan_addr, LOGGER_DATA);
ok *now* shred works with alternate config files
2014-05-08 03:31:21 +00:00
// potential issue here; the xchan_addr points to the primary hub.
// The webbie we were called with may not, so it might not be found
// unless we query for hubloc_addr instead of xchan_addr
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
$r = q("select xchan.*, hubloc.* from xchan
left join hubloc on xchan_hash = hubloc_hash
put timeout code in place but don't turn it on.
2014-11-13 23:34:35 +00:00
where xchan_addr = '%s' and (hubloc_flags & %d) > 0 limit 1",
Try to ensure full paths don't leak into xchan_addr and hubloc_addr
2013-08-26 03:11:49 +00:00
dbesc($xchan_addr),
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
intval(HUBLOC_FLAGS_PRIMARY)
);
if($r) {
$url = $r[0]['hubloc_url'];
sort out some anomalies
2014-08-26 04:42:46 +00:00
if($r[0]['hubloc_network'] && $r[0]['hubloc_network'] !== 'zot') {
logger('zot_finger: alternate network: ' . $webbie);
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
logger('url: '.$url.', net: '.var_export($r[0]['hubloc_network'],true), LOGGER_DATA);
sort out some anomalies
2014-08-26 04:42:46 +00:00
return array('success' => false);
}
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
}
else {
$url = 'https://' . $host;
}
sort out some anomalies
2014-08-26 04:42:46 +00:00
rework a couple of zot basics
2012-11-11 07:26:12 +00:00
$rhs = '/.well-known/zot-info';
use our own CA bundle as authoritative for backend communications. This avoids OS dependent CA validity mismatches.
2013-02-14 04:09:30 +00:00
$https = ((strpos($url,'https://') === 0) ? true : false);
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
improve logging zot_finger
2013-07-22 04:54:20 +00:00
logger('zot_finger: ' . $address . ' at ' . $url, LOGGER_DEBUG);
implement "finger" command
2013-01-12 23:55:37 +00:00
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
if($channel) {
$postvars = array(
add key passing and verification to targeted discovery
2012-11-02 22:34:35 +00:00
'address' => $address,
'target' => $channel['channel_guid'],
'target_sig' => $channel['channel_guid_sig'],
'key' => $channel['channel_pubkey']
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
);
implement "finger" command
2013-01-12 23:55:37 +00:00
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
$result = z_post_url($url . $rhs,$postvars);
use our own CA bundle as authoritative for backend communications. This avoids OS dependent CA validity mismatches.
2013-02-14 04:09:30 +00:00
in probe diagnostic - note https failure as diagnostic info - this also changes zot_finger so you can tell it not to fallback. This could be useful in corporate environments that are off the grid or other high security situations (as much as anything could be considered high security which relies on SSL).
2013-10-02 05:20:09 +00:00
if((! $result['success']) && ($autofallback)) {
use our own CA bundle as authoritative for backend communications. This avoids OS dependent CA validity mismatches.
2013-02-14 04:09:30 +00:00
if($https) {
logger('zot_finger: https failed. falling back to http');
$result = z_post_url('http://' . $host . $rhs,$postvars);
}
}
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
}
else {
implement "finger" command
2013-01-12 23:55:37 +00:00
$rhs .= '?f=&address=' . urlencode($address);
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
$result = z_fetch_url($url . $rhs);
in probe diagnostic - note https failure as diagnostic info - this also changes zot_finger so you can tell it not to fallback. This could be useful in corporate environments that are off the grid or other high security situations (as much as anything could be considered high security which relies on SSL).
2013-10-02 05:20:09 +00:00
if((! $result['success']) && ($autofallback)) {
use our own CA bundle as authoritative for backend communications. This avoids OS dependent CA validity mismatches.
2013-02-14 04:09:30 +00:00
if($https) {
logger('zot_finger: https failed. falling back to http');
$result = z_fetch_url('http://' . $host . $rhs);
}
}
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
}
use our own CA bundle as authoritative for backend communications. This avoids OS dependent CA validity mismatches.
2013-02-14 04:09:30 +00:00
if(! $result['success'])
logger('zot_finger: no results');
get permission discovery working - guid sigs were getting truncated in db
2012-11-01 03:53:02 +00:00
return $result;
}
doco
2013-03-26 04:32:12 +00:00
/**
implement a forced directory update mode where we unconditionally create a directory sync packet. This is needed to ensure that monthly directory pings are propagated to other directory servers so they can each prove for themselves whether or not an account is alive or dead. We do not trust other directories to provide us information beyond "look at this entry and decide for yourself" as doing otherwise would invite rogue directory manipulations. As this scheduled update occurs on all channels across all servers, we should also pick up refresh messages from all existing channel clones and these should also propagate out to all directory servers using the same mechanism (though perhaps not at the same time).
2014-02-11 22:19:20 +00:00
* @function: zot_refresh($them, $channel = null, $force = false)
doco
2013-03-26 04:32:12 +00:00
*
doco
2013-11-29 02:46:59 +00:00
* zot_refresh is typically invoked when somebody has changed permissions of a channel and they are notified
* to fetch new permissions via a finger/discovery operation. This may result in a new connection
* (abook entry) being added to a local channel and it may result in auto-permissions being granted.
*
* Friending in zot is accomplished by sending a refresh packet to a specific channel which indicates a
* permission change has been made by the sender which affects the target channel. The hub controlling
* the target channel does targetted discovery (a zot-finger request requesting permissions for the local
* channel). These are decoded here, and if necessary and abook structure (addressbook) is created to store
* the permissions assigned to this channel.
*
* Initially these abook structures are created with a 'pending' flag, so that no reverse permissions are
* implied until this is approved by the owner channel. A channel can also auto-populate permissions in
* return and send back a refresh packet of its own. This is used by forum and group communication channels
* so that friending and membership in the channel's "club" is automatic.
*
* @param array $them => xchan structure of sender
* @param array $channel => local channel structure of target recipient, required for "friending" operations
doco
2013-03-26 04:32:12 +00:00
*
doco
2013-11-29 02:46:59 +00:00
* @returns boolean true if successful, else false
doco
2013-03-26 04:32:12 +00:00
*/
implement a forced directory update mode where we unconditionally create a directory sync packet. This is needed to ensure that monthly directory pings are propagated to other directory servers so they can each prove for themselves whether or not an account is alive or dead. We do not trust other directories to provide us information beyond "look at this entry and decide for yourself" as doing otherwise would invite rogue directory manipulations. As this scheduled update occurs on all channels across all servers, we should also pick up refresh messages from all existing channel clones and these should also propagate out to all directory servers using the same mechanism (though perhaps not at the same time).
2014-02-11 22:19:20 +00:00
function zot_refresh($them,$channel = null, $force = false) {
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
sort out some anomalies
2014-08-26 04:42:46 +00:00
if(array_key_exists('xchan_network',$them) && ($them['xchan_network'] !== 'zot')) {
logger('zot_refresh: not got zot. ' . $them['xchan_name']);
return true;
}
more logging, some debugging of group membership page
2012-12-30 23:28:13 +00:00
logger('zot_refresh: them: ' . print_r($them,true), LOGGER_DATA);
issue with abook creation
2013-01-03 09:53:22 +00:00
if($channel)
logger('zot_refresh: channel: ' . print_r($channel,true), LOGGER_DATA);
more logging, some debugging of group membership page
2012-12-30 23:28:13 +00:00
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
if($them['hubloc_url'])
$url = $them['hubloc_url'];
else {
put timeout code in place but don't turn it on.
2014-11-13 23:34:35 +00:00
$r = q("select hubloc_url from hubloc where hubloc_hash = '%s' and ( hubloc_flags & %d ) > 0 limit 1",
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
dbesc($them['xchan_hash']),
intval(HUBLOC_FLAGS_PRIMARY)
);
if($r)
$url = $r[0]['hubloc_url'];
}
more logging, some debugging of group membership page
2012-12-30 23:28:13 +00:00
if(! $url) {
logger('zot_refresh: no url');
more progress on onepoll
2012-12-31 06:41:53 +00:00
return false;
more logging, some debugging of group membership page
2012-12-30 23:28:13 +00:00
}
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
one could say we've sort of got zot - at least there are two-way communications for channel meta info, don't yet know if it's working
2012-11-13 00:16:37 +00:00
$postvars = array();
if($channel) {
$postvars['target'] = $channel['channel_guid'];
$postvars['target_sig'] = $channel['channel_guid_sig'];
$postvars['key'] = $channel['channel_pubkey'];
}
if(array_key_exists('xchan_addr',$them) && $them['xchan_addr'])
$postvars['address'] = $them['xchan_addr'];
if(array_key_exists('xchan_hash',$them) && $them['xchan_hash'])
$postvars['guid_hash'] = $them['xchan_hash'];
if(array_key_exists('xchan_guid',$them) && $them['xchan_guid']
&& array_key_exists('xchan_guid_sig',$them) && $them['xchan_guid_sig']) {
$postvars['guid'] = $them['xchan_guid'];
$postvars['guid_sig'] = $them['xchan_guid_sig'];
}
rework a couple of zot basics
2012-11-11 07:26:12 +00:00
$rhs = '/.well-known/zot-info';
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
$result = z_post_url($url . $rhs,$postvars);
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('zot_refresh: zot-info: ' . print_r($result,true), LOGGER_DATA);
more debugging - looks like michameer.dyndns.org is throwing 500s so cannot test further
2013-01-04 00:39:22 +00:00
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
if($result['success']) {
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
$j = json_decode($result['body'],true);
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
searchable flag logic reversed
2013-02-02 11:11:23 +00:00
if(! (($j) && ($j['success']))) {
logger('zot_refresh: result not decodable');
better handling of failures to lookup remote authors
2013-01-26 23:50:31 +00:00
return false;
searchable flag logic reversed
2013-02-02 11:11:23 +00:00
}
better handling of failures to lookup remote authors
2013-01-26 23:50:31 +00:00
directory sync issues
2014-03-04 05:00:42 +00:00
$x = import_xchan($j,(($force) ? UPDATE_FLAGS_FORCED : UPDATE_FLAGS_UPDATED));
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
better handling of failures to lookup remote authors
2013-01-26 23:50:31 +00:00
if(! $x['success'])
return false;
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
$their_perms = 0;
one could say we've sort of got zot - at least there are two-way communications for channel meta info, don't yet know if it's working
2012-11-13 00:16:37 +00:00
if($channel) {
$global_perms = get_perms();
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
if($j['permissions']['data']) {
Protocol: now set data['alg'] on all encapsulated encrypted packets, so that we can more easily retire 'aes256cbc' once it is no longer viable.
2013-11-20 23:20:12 +00:00
$permissions = crypto_unencapsulate(array(
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
'data' => $j['permissions']['data'],
'key' => $j['permissions']['key'],
'iv' => $j['permissions']['iv']),
one could say we've sort of got zot - at least there are two-way communications for channel meta info, don't yet know if it's working
2012-11-13 00:16:37 +00:00
$channel['channel_prvkey']);
if($permissions)
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
$permissions = json_decode($permissions,true);
one could say we've sort of got zot - at least there are two-way communications for channel meta info, don't yet know if it's working
2012-11-13 00:16:37 +00:00
logger('decrypted permissions: ' . print_r($permissions,true), LOGGER_DATA);
}
else
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
$permissions = $j['permissions'];
one could say we've sort of got zot - at least there are two-way communications for channel meta info, don't yet know if it's working
2012-11-13 00:16:37 +00:00
some anomolies found when viewing connections in various cases.
2014-03-03 03:40:59 +00:00
$connected_set = false;
foreach issue
2013-01-02 09:44:50 +00:00
if($permissions && is_array($permissions)) {
foreach($permissions as $k => $v) {
some anomolies found when viewing connections in various cases.
2014-03-03 03:40:59 +00:00
// The connected permission means you are in their address book
if($k === 'connected') {
$connected_set = intval($v);
continue;
}
ensure that we don't have permission settings issues due to version skew across sites.
2014-07-18 03:59:58 +00:00
if(($v) && (array_key_exists($k,$global_perms))) {
foreach issue
2013-01-02 09:44:50 +00:00
$their_perms = $their_perms | intval($global_perms[$k][1]);
}
one could say we've sort of got zot - at least there are two-way communications for channel meta info, don't yet know if it's working
2012-11-13 00:16:37 +00:00
}
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
}
add pending check on group queries
2012-11-30 07:06:03 +00:00
put timeout code in place but don't turn it on.
2014-11-13 23:34:35 +00:00
$r = q("select * from abook where abook_xchan = '%s' and abook_channel = %d and not (abook_flags & %d) > 0 limit 1",
ensure we update the correct abook record
2012-11-13 09:42:08 +00:00
dbesc($x['hash']),
intval($channel['channel_id']),
auto-friending for forums and stuff is almost trivial now. But we will need a page to set the permissions on the "self" abook record. These will be the default permissions for anybody we wish to connect with.
2012-12-02 06:37:41 +00:00
intval(ABOOK_FLAG_SELF)
one could say we've sort of got zot - at least there are two-way communications for channel meta info, don't yet know if it's working
2012-11-13 00:16:37 +00:00
);
this should fix the auto-friend stuff
2013-02-07 22:49:09 +00:00
bring back birthdays
2014-06-03 00:49:19 +00:00
if(array_key_exists('profile',$j) && array_key_exists('next_birthday',$j['profile'])) {
$next_birthday = datetime_convert('UTC','UTC',$j['profile']['next_birthday']);
}
else {
This is long overdue - use a symblic constant NULL_DATE instead of the easily mis-typed sequence '0000-00-00 00:00:00'
2014-09-09 03:35:15 +00:00
$next_birthday = NULL_DATE;
bring back birthdays
2014-06-03 00:49:19 +00:00
}
change flags for one-way connections from hidden to unconnected so we can still use hidden for - well hiding connections
2014-03-04 22:35:42 +00:00
if($r) {
duplicate birthday events showing up
2014-06-05 04:02:57 +00:00
// if the dob is the same as what we have stored (disregarding the year), keep the one
// we have as we may have updated the year after sending a notification; and resetting
// to the one we just received would cause us to create duplicated events.
if(substr($r[0]['abook_dob'],5) == substr($next_birthday,5))
$next_birthday = $r[0]['abook_dob'];
change flags for one-way connections from hidden to unconnected so we can still use hidden for - well hiding connections
2014-03-04 22:35:42 +00:00
$current_abook_connected = (($r[0]['abook_flags'] & ABOOK_FLAG_UNCONNECTED) ? 0 : 1);
bring back birthdays
2014-06-03 00:49:19 +00:00
$y = q("update abook set abook_their_perms = %d, abook_dob = '%s'
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
where abook_xchan = '%s' and abook_channel = %d
put timeout code in place but don't turn it on.
2014-11-13 23:34:35 +00:00
and not (abook_flags & %d) > 0 ",
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
intval($their_perms),
bring back birthdays
2014-06-03 00:49:19 +00:00
dbesc($next_birthday),
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
dbesc($x['hash']),
intval($channel['channel_id']),
auto-friending for forums and stuff is almost trivial now. But we will need a page to set the permissions on the "self" abook record. These will be the default permissions for anybody we wish to connect with.
2012-12-02 06:37:41 +00:00
intval(ABOOK_FLAG_SELF)
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
);
some anomolies found when viewing connections in various cases.
2014-03-03 03:40:59 +00:00
bring friends back
2014-03-06 09:29:42 +00:00
// if(($connected_set === 0 || $connected_set === 1) && ($connected_set !== $current_abook_unconnected)) {
change flags for one-way connections from hidden to unconnected so we can still use hidden for - well hiding connections
2014-03-04 22:35:42 +00:00
// if they are in your address book but you aren't in theirs, and/or this does not
// match your current connected state setting, toggle it.
put timeout code in place but don't turn it on.
2014-11-13 23:34:35 +00:00
// FIXME: uncoverted to postgres
// FIXME: when this was enabled, all contacts became unconnected. Currently disabled intentionally
bring friends back
2014-03-06 09:29:42 +00:00
// $y1 = q("update abook set abook_flags = (abook_flags ^ %d)
// where abook_xchan = '%s' and abook_channel = %d
// and not (abook_flags & %d) limit 1",
// intval(ABOOK_FLAG_UNCONNECTED),
// dbesc($x['hash']),
// intval($channel['channel_id']),
// intval(ABOOK_FLAG_SELF)
// );
// }
some anomolies found when viewing connections in various cases.
2014-03-03 03:40:59 +00:00
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
if(! $y)
logger('abook update failed');
populate some posts when somebody is granted "read stream" permission
2013-11-18 00:50:32 +00:00
else {
// if we were just granted read stream permission and didn't have it before, try to pull in some posts
if((! ($r[0]['abook_their_perms'] & PERMS_R_STREAM)) && ($their_perms & PERMS_R_STREAM))
proc_run('php','include/onepoll.php',$r[0]['abook_id']);
}
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
}
else {
ok heads up - potentially destabilising change. I've tried to sort out all the default connection permissions for those who don't have a predefined (or therefore have a "custom") permissions role. Unfortunately this includes most people that were using this software more than a month ago. The real changes are that the SELF address book entry no longer holds "auto-permissions" but instead holds your "default permissions" (if you have a pre-defined role, the defaults will be pulled from the role table). The auto permissions have moved to a pconfig (uid.system.autoperms). A DB update will move these settings into their new homes. What used to be the "Auto-permissions settings" page is now the "default permissions settings" page and a checkbox therein decides whether or not to apply the permissions automatically. A link to this page will only be shown when you have the "custom" role selected. With luck nobody will notice anything wrong. But at least for the next few days, please review permissions that have been assigned to new connections (either automatically or manually) and make sure they make sense (e.g. they aren't "nothing"). You still need to take action when seeing a message "permissions have changed but not yet submitted" as we always let you review and perhaps adjust the settings _before_ a connection is established (unless you have autoperms turned on).
2014-11-05 01:11:02 +00:00
$role = get_pconfig($channel['channel_id'],'system','permissions_role');
if($role) {
don't overwrite $x (contains import_xchan result) with role permissions
2014-11-06 16:24:05 +00:00
$xx = get_role_perms($role);
if($xx['perms_auto'])
$default_perms = $xx['perms_accept'];
ok heads up - potentially destabilising change. I've tried to sort out all the default connection permissions for those who don't have a predefined (or therefore have a "custom") permissions role. Unfortunately this includes most people that were using this software more than a month ago. The real changes are that the SELF address book entry no longer holds "auto-permissions" but instead holds your "default permissions" (if you have a pre-defined role, the defaults will be pulled from the role table). The auto permissions have moved to a pconfig (uid.system.autoperms). A DB update will move these settings into their new homes. What used to be the "Auto-permissions settings" page is now the "default permissions settings" page and a checkbox therein decides whether or not to apply the permissions automatically. A link to this page will only be shown when you have the "custom" role selected. With luck nobody will notice anything wrong. But at least for the next few days, please review permissions that have been assigned to new connections (either automatically or manually) and make sure they make sense (e.g. they aren't "nothing"). You still need to take action when seeing a message "permissions have changed but not yet submitted" as we always let you review and perhaps adjust the settings _before_ a connection is established (unless you have autoperms turned on).
2014-11-05 01:11:02 +00:00
}
if(! $default_perms)
$default_perms = intval(get_pconfig($channel['channel_id'],'system','autoperms'));
auto-friending for forums and stuff is almost trivial now. But we will need a page to set the permissions on the "self" abook record. These will be the default permissions for anybody we wish to connect with.
2012-12-02 06:37:41 +00:00
Implements a new permission level PERMS_PENDING With this a user can allow some action to any user which connects to them, even before they've connected back. Ref. https://mobiliza.org.br/display/478d9e71eaf55748dc646d3990651d6d34cfb7db5c38360538ec730ca3ccf908@zothub.com Also some code cleanup and an alternative logic for handling notifications of permission changes in zot.php. This assumes that private posts are still restricted to people in your addressbook. Regardless of your global permissions, a pending channel won't get private posts, even if the post only has a deny clause not matching the pending channel.
2014-06-17 07:26:18 +00:00
// Keep original perms to check if we need to notify them
$previous_perms = get_all_perms($channel['channel_id'],$x['hash']);
bring back birthdays
2014-06-03 00:49:19 +00:00
$y = q("insert into abook ( abook_account, abook_channel, abook_xchan, abook_their_perms, abook_my_perms, abook_created, abook_updated, abook_dob, abook_flags ) values ( %d, %d, '%s', %d, %d, '%s', '%s', '%s', %d )",
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
intval($channel['channel_account_id']),
intval($channel['channel_id']),
dbesc($x['hash']),
intval($their_perms),
auto-friending for forums and stuff is almost trivial now. But we will need a page to set the permissions on the "self" abook record. These will be the default permissions for anybody we wish to connect with.
2012-12-02 06:37:41 +00:00
intval($default_perms),
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
dbesc(datetime_convert()),
dbesc(datetime_convert()),
bring back birthdays
2014-06-03 00:49:19 +00:00
dbesc($next_birthday),
set_baseurl issue, more cleanup
2012-12-20 00:48:17 +00:00
intval(($default_perms) ? 0 : ABOOK_FLAG_PENDING)
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
);
this should fix the auto-friend stuff
2013-02-07 22:49:09 +00:00
auto-friending for forums and stuff is almost trivial now. But we will need a page to set the permissions on the "self" abook record. These will be the default permissions for anybody we wish to connect with.
2012-12-02 06:37:41 +00:00
if($y) {
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
logger("New introduction received for {$channel['channel_name']}");
Implements a new permission level PERMS_PENDING With this a user can allow some action to any user which connects to them, even before they've connected back. Ref. https://mobiliza.org.br/display/478d9e71eaf55748dc646d3990651d6d34cfb7db5c38360538ec730ca3ccf908@zothub.com Also some code cleanup and an alternative logic for handling notifications of permission changes in zot.php. This assumes that private posts are still restricted to people in your addressbook. Regardless of your global permissions, a pending channel won't get private posts, even if the post only has a deny clause not matching the pending channel.
2014-06-17 07:26:18 +00:00
$new_perms = get_all_perms($channel['channel_id'],$x['hash']);
if($new_perms != $previous_perms) {
// Send back a permissions update if permissions have changed
put timeout code in place but don't turn it on.
2014-11-13 23:34:35 +00:00
$z = q("select * from abook where abook_xchan = '%s' and abook_channel = %d and not (abook_flags & %d) > 0 limit 1",
auto-friending for forums and stuff is almost trivial now. But we will need a page to set the permissions on the "self" abook record. These will be the default permissions for anybody we wish to connect with.
2012-12-02 06:37:41 +00:00
dbesc($x['hash']),
intval($channel['channel_id']),
intval(ABOOK_FLAG_SELF)
);
if($z)
typo in auto permissions
2013-05-20 03:16:05 +00:00
proc_run('php','include/notifier.php','permission_update',$z[0]['abook_id']);
auto-friending for forums and stuff is almost trivial now. But we will need a page to set the permissions on the "self" abook record. These will be the default permissions for anybody we wish to connect with.
2012-12-02 06:37:41 +00:00
}
pull in some posts when we first connect with a new channel (if allowed to) - if not allowed to, do it if that condition changes
2013-11-18 03:22:24 +00:00
$new_connection = q("select abook_id, abook_flags from abook where abook_channel = %d and abook_xchan = '%s' order by abook_created desc limit 1",
populate some posts when somebody is granted "read stream" permission
2013-11-18 00:50:32 +00:00
intval($channel['channel_id']),
dbesc($x['hash'])
);
send new connection notification even if auto-perms are set
2014-06-13 00:31:42 +00:00
if($new_connection) {
new connection notification
2014-03-28 01:03:19 +00:00
require_once('include/enotify.php');
notification(array(
'type' => NOTIFY_INTRO,
'from_xchan' => $x['hash'],
'to_xchan' => $channel['channel_hash'],
'link' => z_root() . '/connedit/' . $new_connection[0]['abook_id'],
));
}
populate some posts when somebody is granted "read stream" permission
2013-11-18 00:50:32 +00:00
Implements a new permission level PERMS_PENDING With this a user can allow some action to any user which connects to them, even before they've connected back. Ref. https://mobiliza.org.br/display/478d9e71eaf55748dc646d3990651d6d34cfb7db5c38360538ec730ca3ccf908@zothub.com Also some code cleanup and an alternative logic for handling notifications of permission changes in zot.php. This assumes that private posts are still restricted to people in your addressbook. Regardless of your global permissions, a pending channel won't get private posts, even if the post only has a deny clause not matching the pending channel.
2014-06-17 07:26:18 +00:00
if($new_connection && ($their_perms & PERMS_R_STREAM)) {
if(($channel['channel_w_stream'] & PERMS_PENDING)
|| (! ($new_connection[0]['abook_flags'] & ABOOK_FLAG_PENDING)) )
populate some posts when somebody is granted "read stream" permission
2013-11-18 00:50:32 +00:00
proc_run('php','include/onepoll.php',$new_connection[0]['abook_id']);
Implements a new permission level PERMS_PENDING With this a user can allow some action to any user which connects to them, even before they've connected back. Ref. https://mobiliza.org.br/display/478d9e71eaf55748dc646d3990651d6d34cfb7db5c38360538ec730ca3ccf908@zothub.com Also some code cleanup and an alternative logic for handling notifications of permission changes in zot.php. This assumes that private posts are still restricted to people in your addressbook. Regardless of your global permissions, a pending channel won't get private posts, even if the post only has a deny clause not matching the pending channel.
2014-06-17 07:26:18 +00:00
}
populate some posts when somebody is granted "read stream" permission
2013-11-18 00:50:32 +00:00
auto-friending for forums and stuff is almost trivial now. But we will need a page to set the permissions on the "self" abook record. These will be the default permissions for anybody we wish to connect with.
2012-12-02 06:37:41 +00:00
}
Now if you follow somebody it will create an abook entry on the other site with blocked,ignored, and pending set. No "friend notifications" yet.
2012-11-30 02:15:13 +00:00
}
starting to get into the hairy parts of zot - identity, location, and permission synchronisation. After this, messaging should be a piece of cake.
2012-11-11 04:08:07 +00:00
}
return true;
}
return false;
}
doco
2013-03-26 04:32:12 +00:00
/**
* @function: zot_gethub
*
* A guid and a url, both signed by the sender, distinguish a known sender at a known location
zot doco
2013-11-29 00:13:09 +00:00
* This function looks these up to see if the channel is known and therefore previously verified.
* If not, we will need to verify it.
*
* @param array $arr
* $arr must contain:
* string $arr['guid'] => guid of conversant
* string $arr['guid_sig'] => guid signed with conversant's private key
* string $arr['url'] => URL of the origination hub of this communication
* string $arr['url_sig'] => URL signed with conversant's private key
*
*
doco
2013-03-26 04:32:12 +00:00
* @returns: array => hubloc record
*/
remember that the main thing is to keep the main thing the main thing...
2012-08-01 00:55:27 +00:00
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
function zot_gethub($arr) {
got zot
2012-11-13 04:59:18 +00:00
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
if($arr['guid'] && $arr['guid_sig'] && $arr['url'] && $arr['url_sig']) {
provide the site blacklist everywhere it is needed
2014-08-22 04:07:32 +00:00
$blacklisted = false;
$bl1 = get_config('system','blacklisted_sites');
if(is_array($bl1) && $bl1) {
foreach($bl1 as $bl) {
if($bl && strpos($arr['url'],$bl) !== false) {
$blacklisted = true;
break;
}
}
}
if($blacklisted) {
logger('zot_gethub: blacklisted site: ' . $arr['url']);
return null;
}
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
$r = q("select * from hubloc
a few minor changes
2012-08-27 06:05:00 +00:00
where hubloc_guid = '%s' and hubloc_guid_sig = '%s'
and hubloc_url = '%s' and hubloc_url_sig = '%s'
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
limit 1",
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
dbesc($arr['guid']),
dbesc($arr['guid_sig']),
dbesc($arr['url']),
dbesc($arr['url_sig'])
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
);
fix the contorted update logic on the network page
2013-01-05 03:24:17 +00:00
if($r && count($r)) {
logger('zot_gethub: found', LOGGER_DEBUG);
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
return $r[0];
fix the contorted update logic on the network page
2013-01-05 03:24:17 +00:00
}
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
}
more error checking
2013-07-15 03:42:06 +00:00
logger('zot_gethub: not found: ' . print_r($arr,true), LOGGER_DEBUG);
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
return null;
}
zot doco
2013-11-29 00:13:09 +00:00
/**
* @function zot_register_hub($arr)
*
* A communication has been received which has an unknown (to us) sender.
* Perform discovery based on our calculated hash of the sender at the origination address.
* This will fetch the discovery packet of the sender, which contains the public key we
* need to verify our guid and url signatures.
*
* @param array $arr
* $arr must contain:
* string $arr['guid'] => guid of conversant
* string $arr['guid_sig'] => guid signed with conversant's private key
* string $arr['url'] => URL of the origination hub of this communication
* string $arr['url_sig'] => URL signed with conversant's private key
*
*
* @returns array => 'success' (boolean true or false)
* 'message' (optional error string only if success is false)
*/
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
function zot_register_hub($arr) {
rework a couple of zot basics
2012-11-11 07:26:12 +00:00
$result = array('success' => false);
no hub - well doh, no wonder; we were looking for array['hub'] and it's actually array['url'].
2012-12-30 07:45:06 +00:00
if($arr['url'] && $arr['url_sig'] && $arr['guid'] && $arr['guid_sig']) {
rework a couple of zot basics
2012-11-11 07:26:12 +00:00
clean up more code duplication
2014-07-15 04:21:24 +00:00
$guid_hash = make_xchan_hash($arr['guid'],$arr['guid_sig']);
rework a couple of zot basics
2012-11-11 07:26:12 +00:00
zome issues registering new hubs
2013-01-03 00:28:47 +00:00
$url = $arr['url'] . '/.well-known/zot-info/?f=&guid_hash=' . $guid_hash;
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('zot_register_hub: ' . $url, LOGGER_DEBUG);
zome issues registering new hubs
2013-01-03 00:28:47 +00:00
$x = z_fetch_url($url);
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('zot_register_hub: ' . print_r($x,true), LOGGER_DATA);
rework a couple of zot basics
2012-11-11 07:26:12 +00:00
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
if($x['success']) {
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
$record = json_decode($x['body'],true);
zot doco
2013-11-29 00:13:09 +00:00
/*
* We now have a key - only continue registration if our signatures are valid
* AND the guid and guid sig in the returned packet match those provided in
* our current communication.
*/
if((rsa_verify($arr['guid'],base64url_decode($arr['guid_sig']),$record['key']))
&& (rsa_verify($arr['url'],base64url_decode($arr['url_sig']),$record['key']))
&& ($arr['guid'] === $record['guid'])
&& ($arr['guid_sig'] === $record['guid_sig'])) {
$c = import_xchan($record);
if($c['success'])
$result['success'] = true;
}
else {
logger('zot_register_hub: failure to verify returned packet.');
}
zot basic comm structure (real basic)
2012-08-10 03:31:06 +00:00
}
}
rework a couple of zot basics
2012-11-11 07:26:12 +00:00
return $result;
basic hub registration
2012-08-22 06:11:27 +00:00
}
grrr... hashes aren't matching.
2012-11-02 05:23:13 +00:00
zot doco
2013-11-29 00:13:09 +00:00
/**
directory sync issues
2014-03-04 05:00:42 +00:00
* @function import_xchan($arr,$ud_flags = UPDATE_FLAGS_UPDATED)
reversed args in update_modtime, incorrect ud_guid and ud_hash
2014-02-11 04:38:26 +00:00
* Takes an associative array of a fetched discovery packet and updates
zot doco
2013-11-29 00:13:09 +00:00
* all internal data structures which need to be updated as a result.
*
* @param array $arr => json_decoded discovery packet
* @param int $ud_flags
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
* Determines whether to create a directory update record if any changes occur, default is UPDATE_FLAGS_UPDATED
directory sync issues
2014-03-04 05:00:42 +00:00
* $ud_flags = UPDATE_FLAGS_FORCED indicates a forced refresh where we unconditionally create a directory update record
implement a forced directory update mode where we unconditionally create a directory sync packet. This is needed to ensure that monthly directory pings are propagated to other directory servers so they can each prove for themselves whether or not an account is alive or dead. We do not trust other directories to provide us information beyond "look at this entry and decide for yourself" as doing otherwise would invite rogue directory manipulations. As this scheduled update occurs on all channels across all servers, we should also pick up refresh messages from all existing channel clones and these should also propagate out to all directory servers using the same mechanism (though perhaps not at the same time).
2014-02-11 22:19:20 +00:00
* this typically occurs once a month for each channel as part of a scheduled ping to notify the directory
* that the channel still exists
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
* @param array $ud_arr
* If set [typically by update_directory_entry()] indicates a specific update table row and more particularly
* contains a particular address (ud_addr) which needs to be updated in that table.
zot doco
2013-11-29 00:13:09 +00:00
*
* @returns array => 'success' (boolean true or false)
* 'message' (optional error string only if success is false)
*/
more infrastructure changes
2012-11-15 03:27:16 +00:00
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
function import_xchan($arr,$ud_flags = UPDATE_FLAGS_UPDATED, $ud_arr = null) {
more infrastructure changes
2012-11-15 03:27:16 +00:00
add hooks to zot-finger and import_xchan
2013-12-04 01:55:10 +00:00
call_hooks('import_xchan', $arr);
more infrastructure changes
2012-11-15 03:27:16 +00:00
$ret = array('success' => false);
start saving site info
2013-07-24 05:33:56 +00:00
$dirmode = intval(get_config('system','directory_mode'));
update
2013-03-21 09:21:44 +00:00
$changed = false;
implement what I hope will now be the server side of directory sync, add viewsrc to item_photo_menu, and log what changed in import_xchan update objects so we can find out why there are so many updates when nothing _obvious_ has changed that should trigger it.
2013-09-15 10:48:43 +00:00
$what = '';
more infrastructure changes
2012-11-15 03:27:16 +00:00
validate input to import_xchan(), as we don't consistently check it prior to entering this function
2013-07-19 01:22:08 +00:00
if(! (is_array($arr) && array_key_exists('success',$arr) && $arr['success'])) {
logger('import_xchan: invalid data packet: ' . print_r($arr,true));
$ret['message'] = t('Invalid data packet');
return $ret;
}
zot logging to figure out why we occasionally don't have an address on xchan_import - I suspect we don't have anything useful at all.
2014-08-13 03:46:33 +00:00
if(! ($arr['guid'] && $arr['guid_sig'])) {
logger('import_xchan: no identity information provided. ' . print_r($arr,true));
return $ret;
}
validate input to import_xchan(), as we don't consistently check it prior to entering this function
2013-07-19 01:22:08 +00:00
clean up more code duplication
2014-07-15 04:21:24 +00:00
$xchan_hash = make_xchan_hash($arr['guid'],$arr['guid_sig']);
pull the plug on the old location sync section of import_xchan - this has now been split off into its own function. Cross fingers because this is core functionality which was tried and true, well tested; and now it's a bit different. Please revert this if the matrix breaks in the next few hours.
2014-09-15 01:09:16 +00:00
$arr['hash'] = $xchan_hash;
more infrastructure changes
2012-11-15 03:27:16 +00:00
$import_photos = false;
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
if(! rsa_verify($arr['guid'],base64url_decode($arr['guid_sig']),$arr['key'])) {
more logging, some debugging of group membership page
2012-12-30 23:28:13 +00:00
logger('import_xchan: Unable to verify channel signature for ' . $arr['address']);
more infrastructure changes
2012-11-15 03:27:16 +00:00
$ret['message'] = t('Unable to verify channel signature');
return $ret;
}
add some debugging - update times are changing much faster than they're supposed to. Log what has changed.
2013-04-30 01:21:23 +00:00
logger('import_xchan: ' . $xchan_hash, LOGGER_DEBUG);
more infrastructure changes
2012-11-15 03:27:16 +00:00
$r = q("select * from xchan where xchan_hash = '%s' limit 1",
dbesc($xchan_hash)
);
implement premium channel discovery
2013-08-22 00:34:04 +00:00
if(! array_key_exists('connect_url', $arr))
$arr['connect_url'] = '';
try to fix bad addresses that got into the network
2013-08-26 03:17:45 +00:00
if(strpos($arr['address'],'/') !== false)
$arr['address'] = substr($arr['address'],0,strpos($arr['address'],'/'));
hidden directory entries
2013-01-22 08:20:25 +00:00
more infrastructure changes
2012-11-15 03:27:16 +00:00
if($r) {
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
if($r[0]['xchan_photo_date'] != $arr['photo_updated'])
hidden directory entries
2013-01-22 08:20:25 +00:00
$import_photos = true;
security fixes related to directory access and sites that are off the grid
2013-01-22 10:56:32 +00:00
// if we import an entry from a site that's not ours and either or both of us is off the grid - hide the entry.
// TODO: check if we're the same directory realm, which would mean we are allowed to see it
$dirmode = get_config('system','directory_mode');
add public forum identification to libzot. No attempt is made to identify other types of forums or weird custom channel permissions. If the channel is auto-accept and taggable, it's a public forum.
2014-11-10 23:21:04 +00:00
if((($arr['site']['directory_mode'] === 'standalone') || ($dirmode & DIRECTORY_MODE_STANDALONE)) && ($arr['site']['url'] != z_root()))
security fixes related to directory access and sites that are off the grid
2013-01-22 10:56:32 +00:00
$arr['searchable'] = false;
add public forum identification to libzot. No attempt is made to identify other types of forums or weird custom channel permissions. If the channel is auto-accept and taggable, it's a public forum.
2014-11-10 23:21:04 +00:00
searchable flag logic reversed
2013-02-02 11:11:23 +00:00
$hidden = (1 - intval($arr['searchable']));
security fixes related to directory access and sites that are off the grid
2013-01-22 10:56:32 +00:00
// Be careful - XCHAN_FLAGS_HIDDEN should evaluate to 1
searchable flag logic reversed
2013-02-02 11:11:23 +00:00
if(($r[0]['xchan_flags'] & XCHAN_FLAGS_HIDDEN) != $hidden)
hidden directory entries
2013-01-22 08:20:25 +00:00
$new_flags = $r[0]['xchan_flags'] ^ XCHAN_FLAGS_HIDDEN;
else
$new_flags = $r[0]['xchan_flags'];
implement premium channel discovery
2013-08-22 00:34:04 +00:00
adult channel setting
2013-09-23 03:38:24 +00:00
$adult = (($r[0]['xchan_flags'] & XCHAN_FLAGS_SELFCENSORED) ? true : false);
$adult_changed = ((intval($adult) != intval($arr['adult_content'])) ? true : false);
if($adult_changed)
$new_flags = $new_flags ^ XCHAN_FLAGS_SELFCENSORED;
this should be better - "xchan deleted" state on remote sites only follows the xchan_flags and not the channel_pageflags; XCHAN_FLAGS_DELETED should only be set if the channel is to be removed from the entire network. As mentioned in a previous commit, channel_pageflags could be set to PAGE_REMOVED but still leave living clones on other sites.
2014-04-07 23:10:43 +00:00
$deleted = (($r[0]['xchan_flags'] & XCHAN_FLAGS_DELETED) ? true : false);
$deleted_changed = ((intval($deleted) != intval($arr['deleted'])) ? true : false);
if($deleted_changed)
$new_flags = $new_flags ^ XCHAN_FLAGS_DELETED;
try to fix bad addresses that got into the network
2013-08-26 03:17:45 +00:00
indicate public forums in directory results (there will be some [possibly considerable] lag time before existing forums are correctly tagged).
2014-11-11 01:44:02 +00:00
$public_forum = (($r[0]['xchan_flags'] & XCHAN_FLAGS_PUBFORUM) ? true : false);
$pubforum_changed = ((intval($public_forum) != intval($arr['public_forum'])) ? true : false);
if($pubforum_changed)
$new_flags = $r[0]['xchan_flags'] ^ XCHAN_FLAGS_PUBFORUM;
update xchans on url change
2013-03-13 00:39:35 +00:00
if(($r[0]['xchan_name_date'] != $arr['name_updated'])
|| ($r[0]['xchan_connurl'] != $arr['connections_url'])
|| ($r[0]['xchan_flags'] != $new_flags)
|| ($r[0]['xchan_addr'] != $arr['address'])
implement premium channel discovery
2013-08-22 00:34:04 +00:00
|| ($r[0]['xchan_follow'] != $arr['follow_url'])
|| ($r[0]['xchan_connpage'] != $arr['connect_url'])
update xchans on url change
2013-03-13 00:39:35 +00:00
|| ($r[0]['xchan_url'] != $arr['url'])) {
implement premium channel discovery
2013-08-22 00:34:04 +00:00
$r = q("update xchan set xchan_name = '%s', xchan_name_date = '%s', xchan_connurl = '%s', xchan_follow = '%s',
xchan_connpage = '%s', xchan_flags = %d,
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
xchan_addr = '%s', xchan_url = '%s' where xchan_hash = '%s'",
don't allow an xchan to be imported with an empty name. To avoid translation issues we'll just make it '-'.
2014-09-12 01:11:07 +00:00
dbesc(($arr['name']) ? $arr['name'] : '-'),
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
dbesc($arr['name_updated']),
more poco discovery
2013-01-01 10:06:41 +00:00
dbesc($arr['connections_url']),
implement premium channel discovery
2013-08-22 00:34:04 +00:00
dbesc($arr['follow_url']),
dbesc($arr['connect_url']),
hidden directory entries
2013-01-22 08:20:25 +00:00
intval($new_flags),
update xchans on url change
2013-03-13 00:39:35 +00:00
dbesc($arr['address']),
dbesc($arr['url']),
more infrastructure changes
2012-11-15 03:27:16 +00:00
dbesc($xchan_hash)
);
add some debugging - update times are changing much faster than they're supposed to. Log what has changed.
2013-04-30 01:21:23 +00:00
logger('import_xchan: existing: ' . print_r($r[0],true), LOGGER_DATA);
logger('import_xchan: new: ' . print_r($arr,true), LOGGER_DATA);
implement what I hope will now be the server side of directory sync, add viewsrc to item_photo_menu, and log what changed in import_xchan update objects so we can find out why there are so many updates when nothing _obvious_ has changed that should trigger it.
2013-09-15 10:48:43 +00:00
$what .= 'xchan ';
update
2013-03-21 09:21:44 +00:00
$changed = true;
more infrastructure changes
2012-11-15 03:27:16 +00:00
}
}
else {
$import_photos = true;
set hidden flags on new directory entries, not just existing ones
2013-03-27 02:37:33 +00:00
if((($arr['site']['directory_mode'] === 'standalone') || ($dirmode & DIRECTORY_MODE_STANDALONE))
&& ($arr['site']['url'] != z_root()))
$arr['searchable'] = false;
$hidden = (1 - intval($arr['searchable']));
if($hidden)
$new_flags = XCHAN_FLAGS_HIDDEN;
else
$new_flags = 0;
adult channel setting
2013-09-23 03:38:24 +00:00
if($arr['adult_content'])
$new_flags |= XCHAN_FLAGS_SELFCENSORED;
this should be better - "xchan deleted" state on remote sites only follows the xchan_flags and not the channel_pageflags; XCHAN_FLAGS_DELETED should only be set if the channel is to be removed from the entire network. As mentioned in a previous commit, channel_pageflags could be set to PAGE_REMOVED but still leave living clones on other sites.
2014-04-07 23:10:43 +00:00
if(array_key_exists('deleted',$arr) && $arr['deleted'])
$new_flags |= XCHAN_FLAGS_DELETED;
set hidden flags on new directory entries, not just existing ones
2013-03-27 02:37:33 +00:00
more infrastructure changes
2012-11-15 03:27:16 +00:00
$x = q("insert into xchan ( xchan_hash, xchan_guid, xchan_guid_sig, xchan_pubkey, xchan_photo_mimetype,
implement premium channel discovery
2013-08-22 00:34:04 +00:00
xchan_photo_l, xchan_addr, xchan_url, xchan_connurl, xchan_follow, xchan_connpage, xchan_name, xchan_network, xchan_photo_date, xchan_name_date, xchan_flags)
values ( '%s', '%s', '%s', '%s' , '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', %d) ",
more infrastructure changes
2012-11-15 03:27:16 +00:00
dbesc($xchan_hash),
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
dbesc($arr['guid']),
dbesc($arr['guid_sig']),
dbesc($arr['key']),
dbesc($arr['photo_mimetype']),
dbesc($arr['photo']),
dbesc($arr['address']),
dbesc($arr['url']),
more poco discovery
2013-01-01 10:06:41 +00:00
dbesc($arr['connections_url']),
implement premium channel discovery
2013-08-22 00:34:04 +00:00
dbesc($arr['follow_url']),
dbesc($arr['connect_url']),
don't allow an xchan to be imported with an empty name. To avoid translation issues we'll just make it '-'.
2014-09-12 01:11:07 +00:00
dbesc(($arr['name']) ? $arr['name'] : '-'),
more infrastructure changes
2012-11-15 03:27:16 +00:00
dbesc('zot'),
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
dbescdate($arr['photo_updated']),
dbescdate($arr['name_updated']),
set hidden flags on new directory entries, not just existing ones
2013-03-27 02:37:33 +00:00
intval($new_flags)
more infrastructure changes
2012-11-15 03:27:16 +00:00
);
some alteration to the way directory sync was originally supposed to work. I'm making this up as I go and not exactly certain where to go next but it makes more sense now and I think the basic idea will actually work. I'll just have to keep making it up until it does work.
2013-09-09 08:05:29 +00:00
implement what I hope will now be the server side of directory sync, add viewsrc to item_photo_menu, and log what changed in import_xchan update objects so we can find out why there are so many updates when nothing _obvious_ has changed that should trigger it.
2013-09-15 10:48:43 +00:00
$what .= 'new_xchan';
update
2013-03-21 09:21:44 +00:00
$changed = true;
more infrastructure changes
2012-11-15 03:27:16 +00:00
}
if($import_photos) {
photo driver abstraction
2013-04-26 03:01:24 +00:00
require_once('include/photo/photo_driver.php');
more infrastructure changes
2012-11-15 03:27:16 +00:00
issue #225
2013-12-06 02:17:16 +00:00
// see if this is a channel clone that's hosted locally - which we treat different from other xchans/connections
$local = q("select channel_account_id, channel_id from channel where channel_hash = '%s' limit 1",
dbesc($xchan_hash)
);
if($local) {
$ph = z_fetch_url($arr['photo'],true);
if($ph['success']) {
import_channel_photo($ph['body'], $arr['photo_mimetype'], $local[0]['channel_account_id'],$local[0]['channel_id']);
// reset the names in case they got messed up when we had a bug in this function
$photos = array(
z_root() . '/photo/profile/l/' . $local[0]['channel_id'],
z_root() . '/photo/profile/m/' . $local[0]['channel_id'],
z_root() . '/photo/profile/s/' . $local[0]['channel_id'],
Often somebody will join the matrix with a dodgy cert and we'll sort that out but their profile photo will remain rainbow man (or the site default). However the photo_date has been updated so we won't try again. This checkin looks for such a failure and leaves the photo_date alone if the photo import failed.
2014-04-14 01:30:06 +00:00
$arr['photo_mimetype'],
false
issue #225
2013-12-06 02:17:16 +00:00
);
}
}
else {
$photos = import_profile_photo($arr['photo'],$xchan_hash);
}
if($photos) {
Often somebody will join the matrix with a dodgy cert and we'll sort that out but their profile photo will remain rainbow man (or the site default). However the photo_date has been updated so we won't try again. This checkin looks for such a failure and leaves the photo_date alone if the photo import failed.
2014-04-14 01:30:06 +00:00
if($photos[4]) {
// importing the photo failed somehow. Leave the photo_date alone so we can try again at a later date.
// This often happens when somebody joins the matrix with a bad cert.
$r = q("update xchan set xchan_photo_l = '%s', xchan_photo_m = '%s', xchan_photo_s = '%s', xchan_photo_mimetype = '%s'
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
where xchan_hash = '%s'",
Often somebody will join the matrix with a dodgy cert and we'll sort that out but their profile photo will remain rainbow man (or the site default). However the photo_date has been updated so we won't try again. This checkin looks for such a failure and leaves the photo_date alone if the photo import failed.
2014-04-14 01:30:06 +00:00
dbesc($photos[0]),
dbesc($photos[1]),
dbesc($photos[2]),
dbesc($photos[3]),
dbesc($xchan_hash)
);
}
else {
$r = q("update xchan set xchan_photo_date = '%s', xchan_photo_l = '%s', xchan_photo_m = '%s', xchan_photo_s = '%s', xchan_photo_mimetype = '%s'
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
where xchan_hash = '%s'",
dbescdate(datetime_convert('UTC','UTC',$arr['photo_updated'])),
Often somebody will join the matrix with a dodgy cert and we'll sort that out but their profile photo will remain rainbow man (or the site default). However the photo_date has been updated so we won't try again. This checkin looks for such a failure and leaves the photo_date alone if the photo import failed.
2014-04-14 01:30:06 +00:00
dbesc($photos[0]),
dbesc($photos[1]),
dbesc($photos[2]),
dbesc($photos[3]),
dbesc($xchan_hash)
);
}
issue #225
2013-12-06 02:17:16 +00:00
$what .= 'photo ';
$changed = true;
}
more infrastructure changes
2012-11-15 03:27:16 +00:00
}
.
2013-03-11 01:45:58 +00:00
// what we are missing for true hub independence is for any changes in the primary hub to
// get reflected not only in the hublocs, but also to update the URLs and addr in the appropriate xchan
ensure that every imported channel has a primary hubloc
2013-12-03 07:55:57 +00:00
pull the plug on the old location sync section of import_xchan - this has now been split off into its own function. Cross fingers because this is core functionality which was tried and true, well tested; and now it's a bit different. Please revert this if the matrix breaks in the next few hours.
2014-09-15 01:09:16 +00:00
$s = sync_locations($arr,$arr);
believe i found the issue which was causing hundreds/thousands of identical hublocs to be created
2014-02-20 09:35:08 +00:00
pull the plug on the old location sync section of import_xchan - this has now been split off into its own function. Cross fingers because this is core functionality which was tried and true, well tested; and now it's a bit different. Please revert this if the matrix breaks in the next few hours.
2014-09-15 01:09:16 +00:00
if($s) {
if($s['change_message'])
$what .= $s['change_message'];
if($s['changed'])
$changed = $s['changed'];
if($s['message'])
$ret['message'] .= $s['message'];
more infrastructure changes
2012-11-15 03:27:16 +00:00
}
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
// Which entries in the update table are we interested in updating?
cleanup
2014-03-18 23:50:46 +00:00
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
$address = (($ud_arr && $ud_arr['ud_addr']) ? $ud_arr['ud_addr'] : $arr['address']);
cleanup
2014-03-18 23:50:46 +00:00
change the way profiles are imported
2013-05-01 01:16:51 +00:00
// Are we a directory server of some kind?
start saving site info
2013-07-24 05:33:56 +00:00
add collections widget to connections page - and some more work on realms
2014-08-18 00:48:41 +00:00
$other_realm = false;
$realm = get_directory_realm();
if(array_key_exists('site',$arr)
&& array_key_exists('realm',$arr['site'])
realm detection logic reversed
2014-09-04 23:09:52 +00:00
&& (strpos($arr['site']['realm'],$realm) === false))
add collections widget to connections page - and some more work on realms
2014-08-18 00:48:41 +00:00
$other_realm = true;
change the way profiles are imported
2013-05-01 01:16:51 +00:00
if($dirmode != DIRECTORY_MODE_NORMAL) {
add collections widget to connections page - and some more work on realms
2014-08-18 00:48:41 +00:00
// We're some kind of directory server. However we can only add directory information
// if the entry is in the same realm (or is a sub-realm). Sub-realms are denoted by
// including the parent realm in the name. e.g. 'RED_GLOBAL:foo' would allow an entry to
// be in directories for the local realm (foo) and also the RED_GLOBAL realm.
if(array_key_exists('profile',$arr) && is_array($arr['profile']) && (! $other_realm)) {
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
$profile_changed = import_directory_profile($xchan_hash,$arr['profile'],$address,$ud_flags, 1);
change the way profiles are imported
2013-05-01 01:16:51 +00:00
if($profile_changed) {
implement what I hope will now be the server side of directory sync, add viewsrc to item_photo_menu, and log what changed in import_xchan update objects so we can find out why there are so many updates when nothing _obvious_ has changed that should trigger it.
2013-09-15 10:48:43 +00:00
$what .= 'profile ';
change the way profiles are imported
2013-05-01 01:16:51 +00:00
$changed = true;
}
}
else {
logger('import_xchan: profile not available - hiding');
// they may have made it private
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("delete from xprof where xprof_hash = '%s'",
change the way profiles are imported
2013-05-01 01:16:51 +00:00
dbesc($xchan_hash)
);
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("delete from xtag where xtag_hash = '%s'",
change the way profiles are imported
2013-05-01 01:16:51 +00:00
dbesc($xchan_hash)
);
}
}
start saving site info
2013-07-24 05:33:56 +00:00
if(array_key_exists('site',$arr) && is_array($arr['site'])) {
$profile_changed = import_site($arr['site'],$arr['key']);
if($profile_changed) {
implement what I hope will now be the server side of directory sync, add viewsrc to item_photo_menu, and log what changed in import_xchan update objects so we can find out why there are so many updates when nothing _obvious_ has changed that should trigger it.
2013-09-15 10:48:43 +00:00
$what .= 'site ';
start saving site info
2013-07-24 05:33:56 +00:00
$changed = true;
}
}
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
directory sync issues
2014-03-04 05:00:42 +00:00
if(($changed) || ($ud_flags == UPDATE_FLAGS_FORCED)) {
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
$guid = random_string() . '@' . get_app()->get_hostname();
update_modtime($xchan_hash,$guid,$address,$ud_flags);
implement what I hope will now be the server side of directory sync, add viewsrc to item_photo_menu, and log what changed in import_xchan update objects so we can find out why there are so many updates when nothing _obvious_ has changed that should trigger it.
2013-09-15 10:48:43 +00:00
logger('import_xchan: changed: ' . $what,LOGGER_DEBUG);
update
2013-03-21 09:21:44 +00:00
}
update even if no change
2013-10-15 05:20:14 +00:00
elseif(! $ud_flags) {
// nothing changed but we still need to update the updates record
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and not (ud_flags & %d)>0 ",
much better fix to problem yesterday of updates with the wrong ud_addr getting updated. Now we'll pass in the update table row to import_xchan so we know exactly which ud_addr applies. We still need ud_flags passed in separately.
2014-07-01 04:17:19 +00:00
intval(UPDATE_FLAGS_UPDATED),
dbesc($address),
intval(UPDATE_FLAGS_UPDATED)
);
update even if no change
2013-10-15 05:20:14 +00:00
}
update
2013-03-21 09:21:44 +00:00
more infrastructure changes
2012-11-15 03:27:16 +00:00
if(! x($ret,'message')) {
$ret['success'] = true;
$ret['hash'] = $xchan_hash;
}
more logging, some debugging of group membership page
2012-12-30 23:28:13 +00:00
create_sys_channel during setup bypasses our check when creating the first account that there is a site keypair. Esnure that a site keypair is created before any sys channels are created.
2014-03-24 01:12:32 +00:00
more logging, some debugging of group membership page
2012-12-30 23:28:13 +00:00
logger('import_xchan: result: ' . print_r($ret,true), LOGGER_DATA);
more infrastructure changes
2012-11-15 03:27:16 +00:00
return $ret;
}
zot doco
2013-11-29 00:13:09 +00:00
/**
* @function zot_process_response($hub,$arr,$outq) {
* Called immediately after sending a zot message which is using queue processing
* Updates the queue item according to the response result and logs any information
* returned to aid communications troubleshooting.
*
* @param string $hub - url of site we just contacted
* @param array $arr - output of z_post_url()
* @param array $outq - The queue structure attached to this request
*
* @returns nothing
*/
more delivery diagnostics
2012-12-31 22:45:26 +00:00
function zot_process_response($hub,$arr,$outq) {
if(! $arr['success']) {
logger('zot_process_response: failed: ' . $hub);
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
return;
more delivery diagnostics
2012-12-31 22:45:26 +00:00
}
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
$x = json_decode($arr['body'],true);
more delivery diagnostics
2012-12-31 22:45:26 +00:00
if(! $x) {
logger('zot_process_response: No json from ' . $hub);
logger('zot_process_response: headers: ' . print_r($arr['header'],true), LOGGER_DATA);
}
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
// synchronous message types are handled immediately
// async messages remain in the queue until processed.
if(intval($outq['outq_async'])) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("update outq set outq_delivered = 1, outq_updated = '%s' where outq_hash = '%s' and outq_channel = %d",
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
dbesc(datetime_convert()),
dbesc($outq['outq_hash']),
intval($outq['outq_channel'])
);
}
else {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("delete from outq where outq_hash = '%s' and outq_channel = %d",
DATABASE: outq additions, delivery now getting the notify across to the other side
2012-11-20 00:51:15 +00:00
dbesc($outq['outq_hash']),
intval($outq['outq_channel'])
);
}
logger('zot_process_response: ' . print_r($x,true), LOGGER_DATA);
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
}
doco
2013-03-26 04:32:12 +00:00
/**
zot doco
2013-11-29 00:13:09 +00:00
* @function zot_fetch($arr)
doco
2013-03-26 04:32:12 +00:00
*
zot doco
2013-11-29 00:13:09 +00:00
* We received a notification packet (in mod/post.php) that a message is waiting for us, and we've verified the sender.
* Now send back a pickup message, using our message tracking ID ($arr['secret']), which we will sign with our site private key.
* The entire pickup message is encrypted with the remote site's public key.
* If everything checks out on the remote end, we will receive back a packet containing one or more messages,
* which will be processed and delivered before this function ultimately returns.
*
* @param array $arr
* decrypted and json decoded notify packet from remote site
doco
2013-03-26 04:32:12 +00:00
*/
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
function zot_fetch($arr) {
logger('zot_fetch: ' . print_r($arr,true), LOGGER_DATA);
$url = $arr['sender']['url'] . $arr['callback'];
$ret_hub = zot_gethub($arr['sender']);
if(! $ret_hub) {
doco
2013-03-26 04:32:12 +00:00
logger('zot_fetch: no hub: ' . print_r($arr['sender'],true));
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
return;
}
$data = array(
'type' => 'pickup',
'url' => z_root(),
'callback_sig' => base64url_encode(rsa_sign(z_root() . '/post',get_config('system','prvkey'))),
'callback' => z_root() . '/post',
'secret' => $arr['secret'],
'secret_sig' => base64url_encode(rsa_sign($arr['secret'],get_config('system','prvkey')))
);
Protocol: now set data['alg'] on all encapsulated encrypted packets, so that we can more easily retire 'aes256cbc' once it is no longer viable.
2013-11-20 23:20:12 +00:00
$datatosend = json_encode(crypto_encapsulate(json_encode($data),$ret_hub['hubloc_sitekey']));
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
$fetch = zot_zot($url,$datatosend);
block forgeries
2013-10-02 09:50:02 +00:00
$result = zot_import($fetch, $arr['sender']['url']);
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
return $result;
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
}
more doco
2013-03-27 02:53:04 +00:00
/**
* @function zot_import
*
* Process an incoming array of messages which were obtained via pickup, and
* import, update, delete as directed.
*
zot doco
2013-11-29 00:13:09 +00:00
* @param array $arr => 'pickup' structure returned from remote site
* @param string $sender_url => the url specified by the sender in the initial communication
* we will verify the sender and url in each returned message structure and also verify
* that all the messages returned match the site url that we are currently processing.
*
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
* The message types handled here are 'activity' (e.g. posts), 'mail' , 'profile', 'location',
* and 'channel_sync'
zot doco
2013-11-29 00:13:09 +00:00
*
* @returns array => array ( [0] => string $channel_hash, [1] => string $delivery_status, [2] => string $address )
* suitable for logging remotely, enumerating the processing results of each message/recipient combination.
*
more doco
2013-03-27 02:53:04 +00:00
*/
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
block forgeries
2013-10-02 09:50:02 +00:00
function zot_import($arr, $sender_url) {
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
$data = json_decode($arr['body'],true);
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
It's really not supposed to be quite this chatty on the network - ah that's why. Oops.
2013-01-29 04:51:37 +00:00
if(! $data) {
logger('zot_import: empty body');
return array();
}
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
if(array_key_exists('iv',$data)) {
Protocol: now set data['alg'] on all encapsulated encrypted packets, so that we can more easily retire 'aes256cbc' once it is no longer viable.
2013-11-20 23:20:12 +00:00
$data = json_decode(crypto_unencapsulate($data,get_config('system','prvkey')),true);
Implements a new permission level PERMS_PENDING With this a user can allow some action to any user which connects to them, even before they've connected back. Ref. https://mobiliza.org.br/display/478d9e71eaf55748dc646d3990651d6d34cfb7db5c38360538ec730ca3ccf908@zothub.com Also some code cleanup and an alternative logic for handling notifications of permission changes in zot.php. This assumes that private posts are still restricted to people in your addressbook. Regardless of your global permissions, a pending channel won't get private posts, even if the post only has a deny clause not matching the pending channel.
2014-06-17 07:26:18 +00:00
}
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
$incoming = $data['pickup'];
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
$return = array();
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
if(is_array($incoming)) {
foreach($incoming as $i) {
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
$result = null;
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
if(array_key_exists('iv',$i['notify'])) {
Protocol: now set data['alg'] on all encapsulated encrypted packets, so that we can more easily retire 'aes256cbc' once it is no longer viable.
2013-11-20 23:20:12 +00:00
$i['notify'] = json_decode(crypto_unencapsulate($i['notify'],get_config('system','prvkey')),true);
php warnings
2014-11-03 03:22:18 +00:00
}
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('zot_import: notify: ' . print_r($i['notify'],true), LOGGER_DATA);
y'all got mail
2012-12-06 00:44:07 +00:00
block forgeries
2013-10-02 09:50:02 +00:00
$hub = zot_gethub($i['notify']['sender']);
if((! $hub) || ($hub['hubloc_url'] != $sender_url)) {
logger('zot_import: potential forgery: wrong site for sender: ' . $sender_url . ' != ' . print_r($i['notify'],true));
continue;
}
flag notify packets that are sent in response to a message_id request and prevent them from recursing
2014-10-31 11:59:30 +00:00
$message_request = ((array_key_exists('message_id',$i['notify'])) ? true : false);
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
if($message_request)
logger('processing message request');
block forgeries
2013-10-02 09:50:02 +00:00
clean up more code duplication
2014-07-15 04:21:24 +00:00
$i['notify']['sender']['hash'] = make_xchan_hash($i['notify']['sender']['guid'],$i['notify']['sender']['guid_sig']);
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
$deliveries = null;
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
if(array_key_exists('recipients',$i['notify']) && count($i['notify']['recipients'])) {
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
logger('specific recipients');
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
$recip_arr = array();
foreach($i['notify']['recipients'] as $recip) {
clean up more code duplication
2014-07-15 04:21:24 +00:00
$recip_arr[] = make_xchan_hash($recip['guid'],$recip['guid_sig']);
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
}
stringify_array_elms($recip_arr);
y'all got mail
2012-12-06 00:44:07 +00:00
$recips = implode(',',$recip_arr);
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select channel_hash as hash from channel where channel_hash in ( " . $recips . " ) and not ( channel_pageflags & %d )>0 ",
several things were not working correctly w/r/t community tagging. The preference vanished from settings at some point, and we also weren't updating the original post timestamp so that the changed taxonomy would propagate correctly as an edit.
2014-02-17 00:04:46 +00:00
intval(PAGE_REMOVED)
);
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
if(! $r) {
logger('recips: no recipients on this site');
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
continue;
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
}
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
public scope delivery issues
2014-08-07 04:16:24 +00:00
// It's a specifically targetted post. If we were sent a public_scope hint (likely),
// get rid of it so that it doesn't get stored and cause trouble.
php warnings
2014-11-03 03:22:18 +00:00
if(($i) && is_array($i) && array_key_exists('message',$i) && is_array($i['message']) && array_key_exists('public_scope',$i['message']))
public scope delivery issues
2014-08-07 04:16:24 +00:00
unset($i['message']['public_scope']);
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
$deliveries = $r;
// We found somebody on this site that's in the recipient list.
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
}
else {
silence a warning
2013-11-15 03:33:28 +00:00
if(($i['message']) && (array_key_exists('flags',$i['message'])) && (in_array('private',$i['message']['flags']))) {
public scope delivery issues
2014-08-07 04:16:24 +00:00
if(array_key_exists('public_scope',$i['message']) && $i['message']['public_scope'] === 'public') {
// This should not happen but until we can stop it...
logger('private message was delivered with no recipients.');
continue;
}
need to figure out what's causing this
2013-09-13 05:50:41 +00:00
}
over-ride the private flag if we received a post that isn't completely public, but they didn't set the private flag.
2014-08-08 00:49:57 +00:00
logger('public post');
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
quick-n-dirty wall-to-wall detection, wasn't working at all - this may be overboard or it may still not work at all.
2013-01-25 04:43:33 +00:00
// Public post. look for any site members who are or may be accepting posts from this sender
public scope
2013-02-15 22:13:58 +00:00
// and who are allowed to see them based on the sender's permissions
$deliveries = allowed_public_recips($i);
possible fix for mention tags not posting to mentioned forum (when send stream permission is blocked)
2013-08-22 12:50:15 +00:00
provide a way to sync locations and get rid of bogus hublocs, now implemented
2014-09-15 05:19:19 +00:00
if($i['message'] && array_key_exists('type',$i['message']) && $i['message']['type'] === 'location') {
$sys = get_sys_channel();
$deliveries = array(array('hash' => $sys['xchan_hash']));
}
over-ride the private flag if we received a post that isn't completely public, but they didn't set the private flag.
2014-08-08 00:49:57 +00:00
// if the scope is anything but 'public' we're going to store it as private regardless
// of the private flag on the post.
if($i['message'] && array_key_exists('public_scope',$i['message'])
got the logic reversed on that one
2014-08-08 01:58:33 +00:00
&& $i['message']['public_scope'] !== 'public') {
over-ride the private flag if we received a post that isn't completely public, but they didn't set the private flag.
2014-08-08 00:49:57 +00:00
if(! array_key_exists('flags',$i['message']))
$i['message']['flags'] = array();
if(! in_array('private',$i['message']['flags']))
$i['message']['flags'][] = 'private';
}
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
}
filter out duplicate recipients on local deliveries. We only need to deliver stuff once to any particular channel.
2013-11-18 04:43:50 +00:00
// Go through the hash array and remove duplicates. array_unique() won't do this because the array is more than one level.
$no_dups = array();
if($deliveries) {
foreach($deliveries as $d) {
if(! in_array($d['hash'],$no_dups))
$no_dups[] = $d['hash'];
}
if($no_dups) {
$deliveries = array();
foreach($no_dups as $n) {
$deliveries[] = array('hash' => $n);
}
}
}
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
if(! $deliveries) {
logger('zot_import: no deliveries on this site');
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
continue;
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
}
filter out duplicate recipients on local deliveries. We only need to deliver stuff once to any particular channel.
2013-11-18 04:43:50 +00:00
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
if($i['message']) {
if($i['message']['type'] === 'activity') {
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
$arr = get_item_elements($i['message']);
encrypt private messages on disk - there are still a couple of places where the text is leaked in the logs during processing.
2013-07-29 04:04:03 +00:00
better handling of failures to lookup remote authors
2013-01-26 23:50:31 +00:00
if(! array_key_exists('created',$arr)) {
logger('Activity rejected: probable failure to lookup author/owner. ' . print_r($i['message'],true));
continue;
}
yhis should fix the privacy leak - as well as non-private replies to private posts
2013-08-01 21:27:40 +00:00
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('Activity received: ' . print_r($arr,true), LOGGER_DATA);
logger('Activity recipients: ' . print_r($deliveries,true), LOGGER_DATA);
local delivery
2012-12-03 04:54:20 +00:00
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
$relay = ((array_key_exists('flags',$i['message']) && in_array('relay',$i['message']['flags'])) ? true : false);
flag notify packets that are sent in response to a message_id request and prevent them from recursing
2014-10-31 11:59:30 +00:00
$result = process_delivery($i['notify']['sender'],$arr,$deliveries,$relay,false,$message_request);
local delivery
2012-12-03 04:54:20 +00:00
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
}
elseif($i['message']['type'] === 'mail') {
y'all got mail
2012-12-06 00:44:07 +00:00
$arr = get_mail_elements($i['message']);
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('Mail received: ' . print_r($arr,true), LOGGER_DATA);
logger('Mail recipients: ' . print_r($deliveries,true), LOGGER_DATA);
y'all got mail
2012-12-06 00:44:07 +00:00
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
$result = process_mail_delivery($i['notify']['sender'],$arr,$deliveries);
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
more but slow progress on deliveries. The b*tch is trying to match permissions
2012-11-26 07:18:11 +00:00
}
profile messages for directory synchronisation
2012-12-22 12:33:32 +00:00
elseif($i['message']['type'] === 'profile') {
$arr = get_profile_elements($i['message']);
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('Profile received: ' . print_r($arr,true), LOGGER_DATA);
logger('Profile recipients: ' . print_r($deliveries,true), LOGGER_DATA);
profile messages for directory synchronisation
2012-12-22 12:33:32 +00:00
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
$result = process_profile_delivery($i['notify']['sender'],$arr,$deliveries);
local delivery
2012-12-03 04:54:20 +00:00
profile messages for directory synchronisation
2012-12-22 12:33:32 +00:00
}
more work on channel sync
2013-06-27 00:31:02 +00:00
elseif($i['message']['type'] === 'channel_sync') {
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
// $arr = get_channelsync_elements($i['message']);
reverting to fancybox for "save to file" selection box until we find a better way - prettyPhoto just isn't working for this and I really need this functionality ATM until we can figure out how to make prettyPhoto work properly here.
2013-04-22 05:12:18 +00:00
more work on channel sync
2013-06-27 00:31:02 +00:00
$arr = $i['message'];
reverting to fancybox for "save to file" selection box until we find a better way - prettyPhoto just isn't working for this and I really need this functionality ATM until we can figure out how to make prettyPhoto work properly here.
2013-04-22 05:12:18 +00:00
more work on channel sync
2013-06-27 00:31:02 +00:00
logger('Channel sync received: ' . print_r($arr,true), LOGGER_DATA);
logger('Channel sync recipients: ' . print_r($deliveries,true), LOGGER_DATA);
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
$result = process_channel_sync_delivery($i['notify']['sender'],$arr,$deliveries);
more work on channel sync
2013-06-27 00:31:02 +00:00
}
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
elseif($i['message']['type'] === 'location') {
$arr = $i['message'];
logger('Location message received: ' . print_r($arr,true), LOGGER_DATA);
provide a way to sync locations and get rid of bogus hublocs, now implemented
2014-09-15 05:19:19 +00:00
logger('Location message recipients: ' . print_r($deliveries,true), LOGGER_DATA);
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
$result = process_location_delivery($i['notify']['sender'],$arr,$deliveries);
}
profile messages for directory synchronisation
2012-12-22 12:33:32 +00:00
}
flag failed auth attempts in DB table hubloc
2013-09-11 11:59:45 +00:00
if($result){
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
$return = array_merge($return,$result);
flag failed auth attempts in DB table hubloc
2013-09-11 11:59:45 +00:00
}
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
}
}
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
return $return;
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
"heavy lifting" - zot messages flowing, now just need to be parsed and stored at the other end.
2012-11-20 04:45:12 +00:00
}
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
// A public message with no listed recipients can be delivered to anybody who
// has PERMS_NETWORK for that type of post, or PERMS_SITE and is one the same
// site, or PERMS_SPECIFIC and the sender is a contact who is granted
// permissions via their connection permissions in the address book.
// Here we take a given message and construct a list of hashes of everybody
// on the site that we should deliver to.
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
function public_recips($msg) {
add sys channel to public recips if scope public
2014-11-19 04:34:47 +00:00
require_once('include/identity.php');
possible fix for mention tags not posting to mentioned forum (when send stream permission is blocked)
2013-08-22 12:50:15 +00:00
$check_mentions = false;
add sys channel to public recips if scope public
2014-11-19 04:34:47 +00:00
$include_sys = false;
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
if($msg['message']['type'] === 'activity') {
add sys channel to public recips if scope public
2014-11-19 04:34:47 +00:00
$include_sys = true;
so public_recips and allowed_public_recips is working so much better than before, but was still not quite right. We seem to be getting all the right results for top-level posts now, but comments aren't getting through on channels for which we've allowed them to send us their stream, but not comment on our posts. The reason is we were seeing if they could comment - and we only need to do that if we own the post. If they own the post, we only need to check if they can send us their stream.
2013-09-13 03:11:56 +00:00
$col = 'channel_w_stream';
$field = PERMS_W_STREAM;
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
if(array_key_exists('flags',$msg['message']) && in_array('thread_parent', $msg['message']['flags'])) {
so public_recips and allowed_public_recips is working so much better than before, but was still not quite right. We seem to be getting all the right results for top-level posts now, but comments aren't getting through on channels for which we've allowed them to send us their stream, but not comment on our posts. The reason is we were seeing if they could comment - and we only need to do that if we own the post. If they own the post, we only need to check if they can send us their stream.
2013-09-13 03:11:56 +00:00
// check mention recipient permissions on top level posts only
possible fix for mention tags not posting to mentioned forum (when send stream permission is blocked)
2013-08-22 12:50:15 +00:00
$check_mentions = true;
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
}
else {
so public_recips and allowed_public_recips is working so much better than before, but was still not quite right. We seem to be getting all the right results for top-level posts now, but comments aren't getting through on channels for which we've allowed them to send us their stream, but not comment on our posts. The reason is we were seeing if they could comment - and we only need to do that if we own the post. If they own the post, we only need to check if they can send us their stream.
2013-09-13 03:11:56 +00:00
// if this is a comment and it wasn't sent by the post owner, check to see who is allowing them to comment.
// We should have one specific recipient and this step shouldn't be needed unless somebody stuffed up their software.
// We may need this step to protect us from bad guys intentionally stuffing up their software.
// If it is sent by the post owner, we don't need to do this. We only need to see who is receiving the
// owner's stream (which was already set above) - as they control the comment permissions
if($msg['notify']['sender']['guid_sig'] != $msg['message']['owner']['guid_sig']) {
$col = 'channel_w_comment';
$field = PERMS_W_COMMENT;
}
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
}
}
elseif($msg['message']['type'] === 'mail') {
$col = 'channel_w_mail';
$field = PERMS_W_MAIL;
}
if(! $col)
return NULL;
possible fix for mention tags not posting to mentioned forum (when send stream permission is blocked)
2013-08-22 12:50:15 +00:00
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
if($msg['notify']['sender']['url'] === z_root())
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$sql = " where (( " . $col . " & " . PERMS_NETWORK . " )>0 or ( " . $col . " & " . PERMS_SITE . " )>0 or ( " . $col . " & " . PERMS_PUBLIC . ")>0) ";
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
else
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$sql = " where (( " . $col . " & " . PERMS_NETWORK . " )>0 or ( " . $col . " & " . PERMS_PUBLIC . ")>0) ";
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
is_commentable() had some major issues when applied to the atrocity known as ConversationObject, hopefully this won't destabilise the network as it's a somewhat major permission tweak related to comments; also add any local clones to allowed_public_recips() as they should always be allowed recipients. Not sure what to do about host permissions in the event of clones. They have more than one host, and it isn't spelled out in the sent message. All of this stuff will make your head hurt.
2013-09-20 12:45:11 +00:00
$r = q("select channel_hash as hash from channel $sql or channel_hash = '%s' ",
dbesc($msg['notify']['sender']['hash'])
);
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
if(! $r)
$r = array();
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$x = q("select channel_hash as hash from channel left join abook on abook_channel = channel_id where abook_xchan = '%s' and not ( channel_pageflags & " . PAGE_REMOVED . " )>0 and (( " . $col . " & " . PERMS_SPECIFIC . " )>0 and ( abook_my_perms & " . $field . " )>0) OR ( " . $col . " & " . PERMS_PENDING . " )>0 OR (( " . $col . " & " . PERMS_CONTACTS . " )>0 and not ( abook_flags & " . ABOOK_FLAG_PENDING . " )>0) ",
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
dbesc($msg['notify']['sender']['hash'])
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
);
if(! $x)
$x = array();
$r = array_merge($r,$x);
possible fix for mention tags not posting to mentioned forum (when send stream permission is blocked)
2013-08-22 12:50:15 +00:00
change logging of some troublesome or noisy bits so we can figure out what is really going on
2014-11-20 03:22:22 +00:00
//logger('message: ' . print_r($msg['message'],true));
there's still a permission issue delivering a public post directly to the sys channel (on the local system), but this puts the local sys channel in the public delivery chain and fixes an issue with unseen counts showing on the discover page (where you can't do anything about it).
2014-11-19 08:56:05 +00:00
add sys channel to public recips if scope public
2014-11-19 04:34:47 +00:00
if($include_sys && array_key_exists('public_scope',$msg['message']) && $msg['message']['public_scope'] === 'public') {
$sys = get_sys_channel();
if($sys)
$r[] = array('hash' => $sys['channel_hash']);
}
possible fix for mention tags not posting to mentioned forum (when send stream permission is blocked)
2013-08-22 12:50:15 +00:00
// look for any public mentions on this site
// They will get filtered by tgroup_check() so we don't need to check permissions now
if($check_mentions && $msg['message']['tags']) {
if(is_array($msg['message']['tags']) && $msg['message']['tags']) {
foreach($msg['message']['tags'] as $tag) {
if(($tag['type'] === 'mention') && (strpos($tag['url'],z_root()) !== false)) {
$address = basename($tag['url']);
if($address) {
$z = q("select channel_hash as hash from channel where channel_address = '%s' limit 1",
dbesc($address)
);
if($z)
$r = array_merge($r,$z);
}
}
}
}
}
more diagnostics
2013-07-15 10:12:51 +00:00
logger('public_recips: ' . print_r($r,true), LOGGER_DATA);
more heavy lifting on delivery - now we can find all the recipients on the system who have any acceptable permissions for receiving a post. Next we just need to process edits and deletes, and otherwise store the message - yay. Still a lot of work, but we've come a long way baby.
2012-11-27 00:48:04 +00:00
return $r;
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
}
public scope
2013-02-15 22:13:58 +00:00
// This is the second part of the above function. We'll find all the channels willing to accept public posts from us,
// then match them against the sender privacy scope and see who in that list that the sender is allowing.
function allowed_public_recips($msg) {
change logging of some troublesome or noisy bits so we can figure out what is really going on
2014-11-20 03:22:22 +00:00
logger('allowed_public_recips: ' . print_r($msg,true),LOGGER_DATA);
public scope
2013-02-15 22:13:58 +00:00
$recips = public_recips($msg);
if(! $recips)
return $recips;
if($msg['message']['type'] === 'mail')
return $recips;
if(array_key_exists('public_scope',$msg['message']))
$scope = $msg['message']['public_scope'];
clean up more code duplication
2014-07-15 04:21:24 +00:00
$hash = make_xchan_hash($msg['notify']['sender']['guid'],$msg['notify']['sender']['guid_sig']);
public scope
2013-02-15 22:13:58 +00:00
don't block "public" messages with authenticated scope
2014-08-15 19:36:24 +00:00
if($scope === 'public' || $scope === 'network: red' || $scope === 'authenticated')
public scope
2013-02-15 22:13:58 +00:00
return $recips;
if(strpos($scope,'site:') === 0) {
if(($scope === 'site: ' . get_app()->get_hostname()) && ($msg['notify']['sender']['url'] === z_root()))
return $recips;
else
return array();
}
if($scope === 'self') {
foreach($recips as $r)
if($r['hash'] === $hash)
return array('hash' => $hash);
}
if($scope === 'contacts') {
$condensed_recips = array();
foreach($recips as $rr)
$condensed_recips[] = $rr['hash'];
$results = array();
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select channel_hash as hash from channel left join abook on abook_channel = channel_id where abook_xchan = '%s' and not ( channel_pageflags & %d )>0 ",
several things were not working correctly w/r/t community tagging. The preference vanished from settings at some point, and we also weren't updating the original post timestamp so that the changed taxonomy would propagate correctly as an edit.
2014-02-17 00:04:46 +00:00
dbesc($hash),
intval(PAGE_REMOVED)
public scope
2013-02-15 22:13:58 +00:00
);
if($r) {
foreach($r as $rr)
if(in_array($rr['hash'],$condensed_recips))
$results[] = array('hash' => $rr['hash']);
}
return $results;
}
return array();
}
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
flag notify packets that are sent in response to a message_id request and prevent them from recursing
2014-10-31 11:59:30 +00:00
function process_delivery($sender,$arr,$deliveries,$relay,$public = false,$request = false) {
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
$result = array();
block forgeries
2013-10-02 09:50:02 +00:00
// We've validated the sender. Now make sure that the sender is the owner or author
more work on firehose
2014-03-27 01:45:01 +00:00
if(! $public) {
if($sender['hash'] != $arr['owner_xchan'] && $sender['hash'] != $arr['author_xchan']) {
logger("process_delivery: sender {$sender['hash']} is not owner {$arr['owner_xchan']} or author {$arr['author_xchan']} - mid {$arr['mid']}");
return;
}
block forgeries
2013-10-02 09:50:02 +00:00
}
can't reset public to false because it may have been passed in as true and this would be wrong and cause even more subtle bugs. Reset to whatever value was passed in.
2014-12-02 02:58:27 +00:00
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
foreach($deliveries as $d) {
can't reset public to false because it may have been passed in as true and this would be wrong and cause even more subtle bugs. Reset to whatever value was passed in.
2014-12-02 02:58:27 +00:00
$local_public = $public;
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
$r = q("select * from channel where channel_hash = '%s' limit 1",
dbesc($d['hash'])
);
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
if(! $r) {
improvements in delivery logging
2013-07-22 00:19:16 +00:00
$result[] = array($d['hash'],'recipients not found');
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
continue;
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
}
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
couple of fixes: - significantly increase the content availability on the discover channel - fix the button group on the blog/list mode which made the border on a single comment button a bit wonky
2014-11-19 22:41:24 +00:00
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
$channel = $r[0];
couple of fixes: - significantly increase the content availability on the discover channel - fix the button group on the blog/list mode which made the border on a single comment button a bit wonky
2014-11-19 22:41:24 +00:00
// allow public postings to the sys channel regardless of permissions
don't import to sys channel from self-censored authors
2014-12-12 08:26:07 +00:00
if(($channel['channel_pageflags'] & PAGE_SYSTEM) && (! $arr['item_private'])) {
can't reset public to false because it may have been passed in as true and this would be wrong and cause even more subtle bugs. Reset to whatever value was passed in.
2014-12-02 02:58:27 +00:00
$local_public = true;
couple of fixes: - significantly increase the content availability on the discover channel - fix the button group on the blog/list mode which made the border on a single comment button a bit wonky
2014-11-19 22:41:24 +00:00
don't import to sys channel from self-censored authors
2014-12-12 08:26:07 +00:00
$r = q("select xchan_flags from xchan where xchan_hash = '%s' limit 1",
dbesc($sender['hash'])
);
// don't import sys channel posts from selfcensored authors
if($r && ($r[0]['xchan_flags'] & XCHAN_FLAGS_SELFCENSORED)) {
$local_public = false;
continue;
}
}
hopefully this won't screw up everything - if it does, revert. Otherwise this should work at delivery time to check tag deliveries and bounce the message before it's stored if the channel doesn't allow you to post and you aren't allowed to tag deliver either. Previously this was handled after the post was already stored so you needed posting permission as well as tag deliver permission to get past the checks.
2013-06-16 07:57:39 +00:00
$tag_delivery = tgroup_check($channel['channel_id'],$arr);
rename 'uri' (and parent_uri) to 'mid' (and parent_mid) since these no longer remotely resemble uri's and are actually message_id's. This change is potentially destabilising because it touches a lot of code and structure. But it has to get done and there's no better time than the present.
2013-03-22 01:25:41 +00:00
$perm = (($arr['mid'] == $arr['parent_mid']) ? 'send_stream' : 'post_comments');
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
fix bug #84, make imported messages from your clones show up on your wall
2013-07-30 00:30:46 +00:00
// This is our own post, possibly coming from a channel clone
if($arr['owner_xchan'] == $d['hash']) {
$arr['item_flags'] = $arr['item_flags'] | ITEM_WALL;
}
else {
// clear the wall flag if it is set
if($arr['item_flags'] & ITEM_WALL) {
$arr['item_flags'] = ($arr['item_flags'] ^ ITEM_WALL);
}
}
can't reset public to false because it may have been passed in as true and this would be wrong and cause even more subtle bugs. Reset to whatever value was passed in.
2014-12-02 02:58:27 +00:00
if((! perm_is_allowed($channel['channel_id'],$sender['hash'],$perm)) && (! $tag_delivery) && (! $local_public)) {
Include the entire original item when notifying other channels of deletions as it contains important routing and scope information. Previously we were only sending a couple of critical fields like the message-id, flags, and creation date. The thinking was that it is deleted, let's not resend the deleted contents anywhere. But in order to route this through the same path the original post took we really need the entire original post with all of its baggage attached.
2014-04-10 23:14:19 +00:00
logger("permission denied for delivery to channel {$channel['channel_id']} {$channel['channel_address']}");
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'permission denied',$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
continue;
}
improvement in source route checking when handling complex *upstream* delivery chains, which were not adequately accounted for in the earlier checkin.
2014-10-10 07:13:46 +00:00
if($arr['mid'] != $arr['parent_mid']) {
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
// check source route.
// We are only going to accept comments from this sender if the comment has the same route as the top-level-post,
// this is so that permissions mismatches between senders apply to the entire conversation
// As a side effect we will also do a preliminary check that we have the top-level-post, otherwise
// processing it is pointless.
$r = q("select route from item where mid = '%s' and uid = %d limit 1",
dbesc($arr['parent_mid']),
intval($channel['channel_id'])
);
if(! $r) {
$result[] = array($d['hash'],'comment parent not found',$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
// We don't seem to have a copy of this conversation or at least the parent
// - so request a copy of the entire conversation to date.
// Don't do this if it's a relay post as we're the ones who are supposed to
// have the copy and we don't want the request to loop.
// Also don't do this if this comment came from a conversation request packet.
// It's possible that comments are allowed but posting isn't and that could
// cause a conversation fetch loop. We can detect these packets since they are
// delivered via a 'notify' packet type that has a message_id element in the
// initial zot packet (just like the corresponding 'request' packet type which
// makes the request).
// We'll also check the send_stream permission - because if it isn't allowed,
// the top level post is unlikely to be imported and
add some more safety checks before turning the conversation request messages back on
2014-11-02 07:06:52 +00:00
// this is just an exercise in futility.
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
can't reset public to false because it may have been passed in as true and this would be wrong and cause even more subtle bugs. Reset to whatever value was passed in.
2014-12-02 02:58:27 +00:00
if((! $relay) && (! $request) && (! $local_public)
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
&& perm_is_allowed($channel['channel_id'],$sender['hash'],'send_stream')) {
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
proc_run('php', 'include/notifier.php', 'request', $channel['channel_id'], $sender['hash'], $arr['parent_mid']);
add some more safety checks before turning the conversation request messages back on
2014-11-02 07:06:52 +00:00
}
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
continue;
}
improvement in source route checking when handling complex *upstream* delivery chains, which were not adequately accounted for in the earlier checkin.
2014-10-10 07:13:46 +00:00
if($relay) {
// reset the route in case it travelled a great distance upstream
// use our parent's route so when we go back downstream we'll match
// with whatever route our parent has.
$arr['route'] = $r[0]['route'];
}
else {
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
improvement in source route checking when handling complex *upstream* delivery chains, which were not adequately accounted for in the earlier checkin.
2014-10-10 07:13:46 +00:00
// going downstream check that we have the same upstream provider that
// sent it to us originally. Ignore it if it came from another source
// (with potentially different permissions)
$current_route = (($arr['route']) ? $arr['route'] . ',' : '') . $sender['hash'];
if($r[0]['route'] != $current_route) {
$result[] = array($d['hash'],'comment route mismatch',$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
continue;
}
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
}
}
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
if($arr['item_restrict'] & ITEM_DELETED) {
issue #65
2013-07-02 02:41:11 +00:00
// remove_community_tag is a no-op if this isn't a community tag activity
remove_community_tag($sender,$arr,$channel['channel_id']);
delete item should now mostly work
2013-01-29 03:24:36 +00:00
$item_id = delete_imported_item($sender,$arr,$channel['channel_id']);
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],(($item_id) ? 'deleted' : 'delete_failed'),$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
delete item should now mostly work
2013-01-29 03:24:36 +00:00
if($relay && $item_id) {
logger('process_delivery: invoking relay');
proc_run('php','include/notifier.php','relay',intval($item_id));
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'relayed',$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
delete item should now mostly work
2013-01-29 03:24:36 +00:00
}
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
continue;
}
don't update locally deleted items
2015-01-06 23:44:23 +00:00
$r = q("select id, edited, item_restrict, item_flags, mid, parent_mid from item where mid = '%s' and uid = %d limit 1",
rename 'uri' (and parent_uri) to 'mid' (and parent_mid) since these no longer remotely resemble uri's and are actually message_id's. This change is potentially destabilising because it touches a lot of code and structure. But it has to get done and there's no better time than the present.
2013-03-22 01:25:41 +00:00
dbesc($arr['mid']),
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
intval($channel['channel_id'])
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
);
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
if($r) {
Tried this once before but had to revert. Basically prevent a comment from being relayed more than once - as it's a huge drain on resources. But last time I tried this, wall-to-wall comments stopped getting relayed. This checkin should do the right thing in both conditions.
2014-10-20 00:31:10 +00:00
// We already have this post.
ah yes, the redundant relay run was for the benefit of wall-to-wall posts issue #633
2014-10-10 22:17:32 +00:00
$item_id = $r[0]['id'];
don't update locally deleted items
2015-01-06 23:44:23 +00:00
if($r[0]['item_restrict'] & ITEM_DELETED) {
// It was deleted locally.
$result[] = array($d['hash'],'update ignored',$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
continue;
}
// Maybe it has been edited?
elseif($arr['edited'] > $r[0]['edited']) {
this might make edits update across the wire, or it might not
2013-03-01 02:20:42 +00:00
$arr['id'] = $r[0]['id'];
edits might really work now
2013-03-01 02:59:03 +00:00
$arr['uid'] = $channel['channel_id'];
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
update_imported_item($sender,$arr,$channel['channel_id']);
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
$result[] = array($d['hash'],'updated',$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
missing source route after edit
2014-10-10 21:45:18 +00:00
if(! $relay)
add_source_route($item_id,$sender['hash']);
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
}
else {
$result[] = array($d['hash'],'update ignored',$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
Tried this once before but had to revert. Basically prevent a comment from being relayed more than once - as it's a huge drain on resources. But last time I tried this, wall-to-wall comments stopped getting relayed. This checkin should do the right thing in both conditions.
2014-10-20 00:31:10 +00:00
// We need this line to ensure wall-to-wall comments are relayed (by falling through to the relay bit),
// and at the same time not relay any other relayable posts more than once, because to do so is very wasteful.
if(! ($r[0]['item_flags'] & ITEM_ORIGIN))
continue;
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
}
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
}
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
else {
$arr['aid'] = $channel['channel_account_id'];
$arr['uid'] = $channel['channel_id'];
provide detailed error to remote site for the myriad of things that can go wrong inside item_store(), !! this changes the return of item_store !!
2013-09-11 02:06:06 +00:00
$item_result = item_store($arr);
lots of little Diaspora issues
2014-08-29 04:49:13 +00:00
$item_id = 0;
if($item_result['success']) {
$item_id = $item_result['item_id'];
$parr = array('item_id' => $item_id,'item' => $arr,'sender' => $sender,'channel' => $channel);
call_hooks('activity_received',$parr);
OK this is important. So read it carefully. This checkin implements route matching of comments so that they are only accepted from the same route as the top-level post they are attached to. This way there should be no mis-match of permissions between any posts in the thread. It may not be completely compatible with comments posted in the past (though I've tried to be, there may be some minor issues). In addition it seems that relaying was invoked more often than necessary - especially when a duplicate post arrived which was not processed because the edited time hadn't changed - it still invoked relaying. This fix should improve site performance considerably for comments cross-posted to forums; which got bounced around a bit and delivered redundantly for no reason. Roll this back *only* if it causes a meltdown or comment loss is "serious" (as in OMG people are dying, make it stop!). If we can get past 24 hours without serious issue we need to get everybody onto this code. There may be some minor comment loss (mostly affecting new comments to older posts or likes of older comments) until the majority of sites have moved to the new code. It may be difficult or impossible to deliver comments to posts that pre-date the addition of source routes (April 1, 2014) to anybody but the top-level post author at his/her primary hub. We may wish to close comments on these posts, but let's see how we go before doing that.
2014-10-10 01:12:53 +00:00
// don't add a source route if it's a relay or later recipients will get a route mismatch
if(! $relay)
add_source_route($item_id,$sender['hash']);
lots of little Diaspora issues
2014-08-29 04:49:13 +00:00
}
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],(($item_id) ? 'posted' : 'storage failed:' . $item_result['message']),$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
}
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
if($relay && $item_id) {
logger('process_delivery: invoking relay');
proc_run('php','include/notifier.php','relay',intval($item_id));
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'relayed',$channel['channel_name'] . ' <' . $channel['channel_address'] . '@' . get_app()->get_hostname() . '>',$arr['mid']);
got comments going back and forth - we really need local_deliver now since even local deliveries are going through zot and that isn't necessary or desirable in the long term
2012-11-28 01:09:01 +00:00
}
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
}
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
if(! $deliveries)
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array('','no recipients','',$arr['mid']);
ensure all delivery result possibilities are logged and communicated to the sender
2013-01-01 08:45:44 +00:00
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('process_delivery: local results: ' . print_r($result,true), LOGGER_DEBUG);
provide delivery report on both sides, local and remote
2013-01-03 10:21:35 +00:00
add individual message delivery reporting
2013-01-01 05:33:11 +00:00
return $result;
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
}
y'all got mail
2012-12-06 00:44:07 +00:00
issue #65
2013-07-02 02:41:11 +00:00
function remove_community_tag($sender,$arr,$uid) {
if(! (activity_match($arr['verb'],ACTIVITY_TAG) && ($arr['obj_type'] == ACTIVITY_OBJ_TAGTERM)))
return;
logger('remove_community_tag: invoked');
if(! get_pconfig($uid,'system','blocktags')) {
logger('remove_community tag: permission denied.');
return;
}
$r = q("select * from item where mid = '%s' and uid = %d limit 1",
dbesc($arr['mid']),
intval($uid)
);
if(! $r) {
logger('remove_community_tag: no item');
return;
}
if(($sender['hash'] != $r[0]['owner_xchan']) && ($sender['hash'] != $r[0]['author_xchan'])) {
logger('remove_community_tag: sender not authorised.');
return;
}
$i = $r[0];
if($i['target'])
convert all stored json calls to json_decode_plus()
2013-08-06 10:54:49 +00:00
$i['target'] = json_decode_plus($i['target']);
issue #65
2013-07-02 02:41:11 +00:00
if($i['object'])
convert all stored json calls to json_decode_plus()
2013-08-06 10:54:49 +00:00
$i['object'] = json_decode_plus($i['object']);
issue #65
2013-07-02 02:41:11 +00:00
if(! ($i['target'] && $i['object'])) {
logger('remove_community_tag: no target/object');
return;
}
$message_id = $i['target']['id'];
$r = q("select id from item where mid = '%s' and uid = %d limit 1",
dbesc($message_id),
intval($uid)
);
if(! $r) {
logger('remove_community_tag: no parent message');
return;
}
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$x = q("delete from term where uid = %d and oid = %d and otype = %d and type = %d and term = '%s' and url = '%s'",
issue #65
2013-07-02 02:41:11 +00:00
intval($uid),
intval($r[0]['id']),
intval(TERM_OBJ_POST),
intval(TERM_HASHTAG),
dbesc($i['object']['title']),
dbesc(get_rel_link($i['object']['link'],'alternate'))
);
return;
}
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
function update_imported_item($sender,$item,$uid) {
this might make edits update across the wire, or it might not
2013-03-01 02:20:42 +00:00
provide detailed error to remote site for the myriad of things that can go wrong inside item_store(), !! this changes the return of item_store !!
2013-09-11 02:06:06 +00:00
$x = item_store_update($item);
if(! $x['item_id'])
logger('update_imported_item: failed: ' . $x['message']);
else
logger('update_imported_item');
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
}
function delete_imported_item($sender,$item,$uid) {
delete item should now mostly work
2013-01-29 03:24:36 +00:00
logger('delete_imported_item invoked',LOGGER_DEBUG);
break delivery loop if an item is deleted twice
2014-01-10 20:37:48 +00:00
$r = q("select id, item_restrict from item where ( author_xchan = '%s' or owner_xchan = '%s' or source_xchan = '%s' )
rename 'uri' (and parent_uri) to 'mid' (and parent_mid) since these no longer remotely resemble uri's and are actually message_id's. This change is potentially destabilising because it touches a lot of code and structure. But it has to get done and there's no better time than the present.
2013-03-22 01:25:41 +00:00
and mid = '%s' and uid = %d limit 1",
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
dbesc($sender['hash']),
move js files from core
2014-01-04 11:58:21 +00:00
dbesc($sender['hash']),
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
dbesc($sender['hash']),
rename 'uri' (and parent_uri) to 'mid' (and parent_mid) since these no longer remotely resemble uri's and are actually message_id's. This change is potentially destabilising because it touches a lot of code and structure. But it has to get done and there's no better time than the present.
2013-03-22 01:25:41 +00:00
dbesc($item['mid']),
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
intval($uid)
);
delete item should now mostly work
2013-01-29 03:24:36 +00:00
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
if(! $r) {
logger('delete_imported_item: failed: ownership issue');
delete item should now mostly work
2013-01-29 03:24:36 +00:00
return false;
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
}
break delivery loop if an item is deleted twice
2014-01-10 20:37:48 +00:00
if($r[0]['item_restrict'] & ITEM_DELETED) {
logger('delete_imported_item: item was already deleted');
return false;
}
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
uncomplicate item deletion and fixup all the other linked tables
2013-01-25 00:44:10 +00:00
require_once('include/items.php');
found what I believe is the root cause of issue #230, but don't yet have a workable solution. This one is hard.
2014-03-21 05:08:16 +00:00
try and sort out the item delete mess
2014-03-28 03:28:48 +00:00
// Use phased deletion to set the deleted flag, call both tag_deliver and the notifier to notify downstream channels
// and then clean up after ourselves with a cron job after several days to do the delete_item_lowlevel() (DROPITEM_PHASE2).
found what I believe is the root cause of issue #230, but don't yet have a workable solution. This one is hard.
2014-03-21 05:08:16 +00:00
try and sort out the item delete mess
2014-03-28 03:28:48 +00:00
drop_item($r[0]['id'],false, DROPITEM_PHASE1);
issue #230 - deletion failure in multiple delivery chains
2013-12-26 08:14:17 +00:00
tag_deliver($uid,$r[0]['id']);
delete item should now mostly work
2013-01-29 03:24:36 +00:00
return $r[0]['id'];
You've got mail! (actually a post, mail is for another day)
2012-11-27 05:20:16 +00:00
}
y'all got mail
2012-12-06 00:44:07 +00:00
function process_mail_delivery($sender,$arr,$deliveries) {
extend delivery reports to mail
2013-05-17 03:43:24 +00:00
$result = array();
post signatures
2013-10-03 04:04:48 +00:00
if($sender['hash'] != $arr['from_xchan']) {
logger('process_mail_delivery: sender is not mail author');
return;
}
y'all got mail
2012-12-06 00:44:07 +00:00
foreach($deliveries as $d) {
$r = q("select * from channel where channel_hash = '%s' limit 1",
dbesc($d['hash'])
);
extend delivery reports to mail
2013-05-17 03:43:24 +00:00
if(! $r) {
$result[] = array($d['hash'],'not found');
y'all got mail
2012-12-06 00:44:07 +00:00
continue;
extend delivery reports to mail
2013-05-17 03:43:24 +00:00
}
y'all got mail
2012-12-06 00:44:07 +00:00
$channel = $r[0];
if(! perm_is_allowed($channel['channel_id'],$sender['hash'],'post_mail')) {
logger("permission denied for mail delivery {$channel['channel_id']}");
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'permission denied',$channel['channel_name'],$arr['mid']);
y'all got mail
2012-12-06 00:44:07 +00:00
continue;
}
rename 'uri' (and parent_uri) to 'mid' (and parent_mid) since these no longer remotely resemble uri's and are actually message_id's. This change is potentially destabilising because it touches a lot of code and structure. But it has to get done and there's no better time than the present.
2013-03-22 01:25:41 +00:00
$r = q("select id from mail where mid = '%s' and channel_id = %d limit 1",
dbesc($arr['mid']),
y'all got mail
2012-12-06 00:44:07 +00:00
intval($channel['channel_id'])
);
if($r) {
mail recall - just set mail_flags = mail_flags & MAIL_RECALLED and tell the notifier. We'll need a special button for this. Also should probably report whether it was seen or not. Will do that another day.
2013-02-28 04:38:33 +00:00
if($arr['mail_flags'] & MAIL_RECALLED) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$x = q("delete from mail where id = %d and channel_id = %d",
mail recall - just set mail_flags = mail_flags & MAIL_RECALLED and tell the notifier. We'll need a special button for this. Also should probably report whether it was seen or not. Will do that another day.
2013-02-28 04:38:33 +00:00
intval($r[0]['id']),
intval($channel['channel_id'])
);
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'mail recalled',$channel['channel_name'],$arr['mid']);
mail recall - just set mail_flags = mail_flags & MAIL_RECALLED and tell the notifier. We'll need a special button for this. Also should probably report whether it was seen or not. Will do that another day.
2013-02-28 04:38:33 +00:00
logger('mail_recalled');
}
else {
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'duplicate mail received',$channel['channel_name'],$arr['mid']);
mail recall - just set mail_flags = mail_flags & MAIL_RECALLED and tell the notifier. We'll need a special button for this. Also should probably report whether it was seen or not. Will do that another day.
2013-02-28 04:38:33 +00:00
logger('duplicate mail received');
}
y'all got mail
2012-12-06 00:44:07 +00:00
continue;
}
else {
$arr['account_id'] = $channel['channel_account_id'];
$arr['channel_id'] = $channel['channel_id'];
$item_id = mail_store($arr);
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'mail delivered',$channel['channel_name'],$arr['mid']);
start rewrite of notification system delivery backend
2013-01-06 02:08:06 +00:00
y'all got mail
2012-12-06 00:44:07 +00:00
}
}
extend delivery reports to mail
2013-05-17 03:43:24 +00:00
return $result;
y'all got mail
2012-12-06 00:44:07 +00:00
}
profile messages for directory synchronisation
2012-12-22 12:33:32 +00:00
function process_profile_delivery($sender,$arr,$deliveries) {
This was bloody hard to sort out.... but now it's sorted and we can move forward again. Put back xtag for efficient keyword searching, use it in conjunction with xprof['keywords'] to avoid a SQL triple join and group_concat (which simply won't scale on a directory server), figured out how best to work timezone corrected birthdays into the protocol. The directory can even provide an age if one was provided to it. Both of these things need some more work, but we have the mechanisms and a plan how to do it right instead of "how the !@#$ are we going to do pull this off?"
2012-12-28 07:33:37 +00:00
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('process_profile_delivery', LOGGER_DEBUG);
populate the new fields in the directory updates table
2013-09-25 03:13:53 +00:00
$r = q("select xchan_addr from xchan where xchan_hash = '%s' limit 1",
dbesc($sender['hash'])
);
if($r)
directory sync issues
2014-03-04 05:00:42 +00:00
import_directory_profile($sender['hash'],$arr,$r[0]['xchan_addr'], UPDATE_FLAGS_UPDATED, 0);
more progress on directory import
2012-12-27 02:18:56 +00:00
}
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
function process_location_delivery($sender,$arr,$deliveries) {
// deliveries is irrelevant
logger('process_location_delivery', LOGGER_DEBUG);
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
$r = q("select xchan_pubkey from xchan where xchan_hash = '%s' limit 1",
dbesc($sender['hash'])
);
if($r)
$sender['key'] = $r[0]['xchan_pubkey'];
new function hubloc_change_primary()
2014-10-13 22:27:03 +00:00
if(array_key_exists('locations',$arr) && $arr['locations']) {
$x = sync_locations($sender,$arr,true);
logger('process_location_delivery: results: ' . print_r($x,true), LOGGER_DEBUG);
if($x['changed']) {
$guid = random_string() . '@' . get_app()->get_hostname();
update_modtime($sender['hash'],$sender['guid'],$arr['locations'][0]['address'],UPDATE_FLAGS_UPDATED);
}
}
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
}
function sync_locations($sender,$arr,$absolute = false) {
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
$ret = array();
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
if($arr['locations']) {
$xisting = q("select hubloc_id, hubloc_url, hubloc_sitekey from hubloc where hubloc_hash = '%s'",
dbesc($sender['hash'])
);
// See if a primary is specified
$has_primary = false;
foreach($arr['locations'] as $location) {
if($location['primary']) {
$has_primary = true;
break;
}
}
cleanup of sync_locations and better logging. There was one significant code change but it is unrelated to current issues, basically if no primary was set we were setting everything as primary.
2014-10-12 22:18:49 +00:00
// Ensure that they have one primary hub
if(! $has_primary)
$arr['locations'][0]['primary'] = true;
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
foreach($arr['locations'] as $location) {
if(! rsa_verify($location['url'],base64url_decode($location['url_sig']),$sender['key'])) {
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
logger('sync_locations: Unable to verify site signature for ' . $location['url']);
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
$ret['message'] .= sprintf( t('Unable to verify site signature for %s'), $location['url']) . EOL;
continue;
}
for($x = 0; $x < count($xisting); $x ++) {
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
if(($xisting[$x]['hubloc_url'] === $location['url'])
&& ($xisting[$x]['hubloc_sitekey'] === $location['sitekey'])) {
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
$xisting[$x]['updated'] = true;
}
}
if(! $location['sitekey']) {
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
logger('sync_locations: empty hubloc sitekey. ' . print_r($location,true));
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
continue;
}
// Catch some malformed entries from the past which still exist
if(strpos($location['address'],'/') !== false)
$location['address'] = substr($location['address'],0,strpos($location['address'],'/'));
// match as many fields as possible in case anything at all changed.
$r = q("select * from hubloc where hubloc_hash = '%s' and hubloc_guid = '%s' and hubloc_guid_sig = '%s' and hubloc_url = '%s' and hubloc_url_sig = '%s' and hubloc_host = '%s' and hubloc_addr = '%s' and hubloc_callback = '%s' and hubloc_sitekey = '%s' ",
dbesc($sender['hash']),
dbesc($sender['guid']),
dbesc($sender['guid_sig']),
dbesc($location['url']),
dbesc($location['url_sig']),
dbesc($location['host']),
dbesc($location['address']),
dbesc($location['callback']),
dbesc($location['sitekey'])
);
if($r) {
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
logger('sync_locations: hub exists: ' . $location['url'], LOGGER_DEBUG);
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
// update connection timestamp if this is the site we're talking to
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
// This only happens when called from import_xchan
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
if(array_key_exists('site',$arr) && $location['url'] == $arr['site']['url']) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update hubloc set hubloc_connected = '%s', hubloc_updated = '%s' where hubloc_id = %d",
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
dbesc(datetime_convert()),
dbesc(datetime_convert()),
intval($r[0]['hubloc_id'])
);
}
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
// if it's marked offline/dead, bring it back
// Should we do this? It's basically saying that the channel knows better than
// the directory server if the site is alive.
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
if($r[0]['hubloc_status'] & HUBLOC_OFFLINE) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update hubloc set hubloc_status = (hubloc_status & ~%d) where hubloc_id = %d",
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
intval(HUBLOC_OFFLINE),
intval($r[0]['hubloc_id'])
);
if($r[0]['hubloc_flags'] & HUBLOC_FLAGS_ORPHANCHECK) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update hubloc set hubloc_flags = (hubloc_flags & ~%d) where hubloc_id = %d",
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
intval(HUBLOC_FLAGS_ORPHANCHECK),
intval($r[0]['hubloc_id'])
);
}
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update xchan set xchan_flags = (xchan_flags & ~%d) where (xchan_flags & %d)>0 and xchan_hash = '%s'",
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
intval(XCHAN_FLAGS_ORPHAN),
intval(XCHAN_FLAGS_ORPHAN),
dbesc($sender['hash'])
);
}
// Remove pure duplicates
if(count($r) > 1) {
for($h = 1; $h < count($r); $h ++) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("delete from hubloc where hubloc_id = %d",
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
intval($r[$h]['hubloc_id'])
);
cleanup of sync_locations and better logging. There was one significant code change but it is unrelated to current issues, basically if no primary was set we were setting everything as primary.
2014-10-12 22:18:49 +00:00
$what .= 'duplicate_hubloc_removed ';
$changed = true;
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
}
}
if((($r[0]['hubloc_flags'] & HUBLOC_FLAGS_PRIMARY) && (! $location['primary']))
|| ((! ($r[0]['hubloc_flags'] & HUBLOC_FLAGS_PRIMARY)) && ($location['primary']))) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$m = q("update hubloc set hubloc_flags = (hubloc_flags & ~%d), hubloc_updated = '%s' where hubloc_id = %d",
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
intval(HUBLOC_FLAGS_PRIMARY),
dbesc(datetime_convert()),
intval($r[0]['hubloc_id'])
);
don't use cached data
2014-10-14 04:01:51 +00:00
// make sure hubloc_change_primary() has current data
$r[0]['hubloc_flags'] = $r[0]['hubloc_flags'] ^ HUBLOC_FLAGS_PRIMARY;
new function hubloc_change_primary()
2014-10-13 22:27:03 +00:00
hubloc_change_primary($r[0]);
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
$what .= 'primary_hub ';
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
$changed = true;
}
figuring out how to bootstrap the change_primary procedure when all you have is inconsistent data which you think you trust.
2014-10-14 04:38:36 +00:00
elseif($absolute) {
// Absolute sync - make sure the current primary is correctly reflected in the xchan
if a hubloc primary change results in an xchan primary change, create a directory update packet.
2014-10-14 04:44:46 +00:00
$pr = hubloc_change_primary($r[0]);
if($pr) {
$what .= 'xchan_primary';
$changed = true;
}
figuring out how to bootstrap the change_primary procedure when all you have is inconsistent data which you think you trust.
2014-10-14 04:38:36 +00:00
}
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
if((($r[0]['hubloc_flags'] & HUBLOC_FLAGS_DELETED) && (! $location['deleted']))
|| ((! ($r[0]['hubloc_flags'] & HUBLOC_FLAGS_DELETED)) && ($location['deleted']))) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$n = q("update hubloc set hubloc_flags = (hubloc_flags & ~%d), hubloc_updated = '%s' where hubloc_id = %d",
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
intval(HUBLOC_FLAGS_DELETED),
dbesc(datetime_convert()),
intval($r[0]['hubloc_id'])
);
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
$what .= 'delete_hub ';
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
$changed = true;
}
continue;
}
cleanup of sync_locations and better logging. There was one significant code change but it is unrelated to current issues, basically if no primary was set we were setting everything as primary.
2014-10-12 22:18:49 +00:00
// Existing hubs are dealt with. Now let's process any new ones.
// New hub claiming to be primary. Make it so by removing any existing primaries.
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
if(intval($location['primary'])) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("update hubloc set hubloc_flags = (hubloc_flags & ~%d), hubloc_updated = '%s' where hubloc_hash = '%s' and (hubloc_flags & %d )>0",
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
intval(HUBLOC_FLAGS_PRIMARY),
dbesc(datetime_convert()),
dbesc($sender['hash']),
intval(HUBLOC_FLAGS_PRIMARY)
);
}
pull the plug on the old location sync section of import_xchan - this has now been split off into its own function. Cross fingers because this is core functionality which was tried and true, well tested; and now it's a bit different. Please revert this if the matrix breaks in the next few hours.
2014-09-15 01:09:16 +00:00
logger('sync_locations: new hub: ' . $location['url']);
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
$r = q("insert into hubloc ( hubloc_guid, hubloc_guid_sig, hubloc_hash, hubloc_addr, hubloc_network, hubloc_flags, hubloc_url, hubloc_url_sig, hubloc_host, hubloc_callback, hubloc_sitekey, hubloc_updated, hubloc_connected)
values ( '%s','%s','%s','%s', '%s', %d ,'%s','%s','%s','%s','%s','%s','%s')",
dbesc($sender['guid']),
dbesc($sender['guid_sig']),
dbesc($sender['hash']),
dbesc($location['address']),
dbesc('zot'),
intval((intval($location['primary'])) ? HUBLOC_FLAGS_PRIMARY : 0),
dbesc($location['url']),
dbesc($location['url_sig']),
dbesc($location['host']),
dbesc($location['callback']),
dbesc($location['sitekey']),
dbesc(datetime_convert()),
dbesc(datetime_convert())
);
$what .= 'newhub ';
$changed = true;
new function hubloc_change_primary()
2014-10-13 22:27:03 +00:00
if($location['primary']) {
$r = q("select * from hubloc where hubloc_addr = '%s' and hubloc_sitekey = '%s' limit 1",
dbesc($location['address']),
dbesc($location['sitekey'])
);
if($r)
hubloc_change_primary($r[0]);
}
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
}
// get rid of any hubs we have for this channel which weren't reported.
if($absolute && $xisting) {
foreach($xisting as $x) {
if(! array_key_exists('updated',$x)) {
just mark dead hubloc deleted - don't remove them. This could cause problems. Also clean up fetch_url/post_url header option
2014-09-17 02:07:19 +00:00
logger('sync_locations: deleting unreferenced hub location ' . $x['hubloc_url']);
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("update hubloc set hubloc_flags = (hubloc_flags & ~%d), hubloc_updated = '%s' where hubloc_id = %d",
just mark dead hubloc deleted - don't remove them. This could cause problems. Also clean up fetch_url/post_url header option
2014-09-17 02:07:19 +00:00
intval(HUBLOC_FLAGS_DELETED),
dbesc(datetime_convert()),
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
intval($x['hubloc_id'])
);
cleanup of sync_locations and better logging. There was one significant code change but it is unrelated to current issues, basically if no primary was set we were setting everything as primary.
2014-10-12 22:18:49 +00:00
$what .= 'removed_hub ';
some backend work for the remaining missing bits of mod_hubman - this is still a fair ways from being complete and is not ready for prime time. Basically we'll let a channel send out a public message saying "these are my currently approved locations" and anything that isn't in the list will be marked deleted. We'll send out this message when locations change somehow - either through direct personal involvement (hub revoke, change primary, channel import) or during a system rename or "find bad/obsolete hublocs" activity. This way we won't have clones sending back location info we just got rid of and re-importing the bad entries.
2014-09-13 23:00:09 +00:00
$changed = true;
}
}
}
}
$ret['change_message'] = $what;
$ret['changed'] = $changed;
return $ret;
}
change the way profiles are imported
2013-05-01 01:16:51 +00:00
more heavy lifting on the hubloc management and revocation infrastructure; don't rush me. This is going to become a fundamental part of zot. It deserves careful consideration.
2014-09-14 08:48:54 +00:00
function zot_encode_locations($channel) {
$ret = array();
$x = zot_get_hublocs($channel['channel_hash']);
if($x && count($x)) {
foreach($x as $hub) {
if(! ($hub['hubloc_flags'] & HUBLOC_FLAGS_UNVERIFIED)) {
$ret[] = array(
'host' => $hub['hubloc_host'],
'address' => $hub['hubloc_addr'],
'primary' => (($hub['hubloc_flags'] & HUBLOC_FLAGS_PRIMARY) ? true : false),
'url' => $hub['hubloc_url'],
'url_sig' => $hub['hubloc_url_sig'],
'callback' => $hub['hubloc_callback'],
'sitekey' => $hub['hubloc_sitekey'],
'deleted' => (($hub['hubloc_flags'] & HUBLOC_FLAGS_DELETED) ? true : false)
);
}
}
}
return $ret;
}
change the way profiles are imported
2013-05-01 01:16:51 +00:00
/*
* @function import_directory_profile
*
* @returns boolean $updated if something changed
*
*/
directory sync issues
2014-03-04 05:00:42 +00:00
function import_directory_profile($hash,$profile,$addr,$ud_flags = UPDATE_FLAGS_UPDATED, $suppress_update = 0) {
more progress on directory import
2012-12-27 02:18:56 +00:00
turn down the log noise a bit
2013-01-25 03:45:08 +00:00
logger('import_directory_profile', LOGGER_DEBUG);
more progress on directory import
2012-12-27 02:18:56 +00:00
if(! $hash)
change the way profiles are imported
2013-05-01 01:16:51 +00:00
return false;
more progress on directory import
2012-12-27 02:18:56 +00:00
$arr = array();
$arr['xprof_hash'] = $hash;
$arr['xprof_dob'] = datetime_convert('','',$profile['birthday'],'Y-m-d'); // !!!! check this for 0000 year
found what I believe is the root cause of issue #230, but don't yet have a workable solution. This one is hard.
2014-03-21 05:08:16 +00:00
$arr['xprof_age'] = (($profile['age']) ? intval($profile['age']) : 0);
$arr['xprof_desc'] = (($profile['description']) ? htmlspecialchars($profile['description'], ENT_COMPAT,'UTF-8',false) : '');
$arr['xprof_gender'] = (($profile['gender']) ? htmlspecialchars($profile['gender'], ENT_COMPAT,'UTF-8',false) : '');
$arr['xprof_marital'] = (($profile['marital']) ? htmlspecialchars($profile['marital'], ENT_COMPAT,'UTF-8',false) : '');
$arr['xprof_sexual'] = (($profile['sexual']) ? htmlspecialchars($profile['sexual'], ENT_COMPAT,'UTF-8',false) : '');
$arr['xprof_locale'] = (($profile['locale']) ? htmlspecialchars($profile['locale'], ENT_COMPAT,'UTF-8',false) : '');
$arr['xprof_region'] = (($profile['region']) ? htmlspecialchars($profile['region'], ENT_COMPAT,'UTF-8',false) : '');
issue #240 - we were using htmlentities instead of htmlspecialchars in several places, and this was a bit greedy in the set of characters which were converted from utf-8 to HTML entities. Also brought mail attachments up to date so they are rendered identically to item attachments.
2013-12-12 07:13:36 +00:00
$arr['xprof_postcode'] = (($profile['postcode']) ? htmlspecialchars($profile['postcode'], ENT_COMPAT,'UTF-8',false) : '');
found what I believe is the root cause of issue #230, but don't yet have a workable solution. This one is hard.
2014-03-21 05:08:16 +00:00
$arr['xprof_country'] = (($profile['country']) ? htmlspecialchars($profile['country'], ENT_COMPAT,'UTF-8',false) : '');
$arr['xprof_about'] = (($profile['about']) ? htmlspecialchars($profile['about'], ENT_COMPAT,'UTF-8',false) : '');
$arr['xprof_homepage'] = (($profile['homepage']) ? htmlspecialchars($profile['homepage'], ENT_COMPAT,'UTF-8',false) : '');
$arr['xprof_hometown'] = (($profile['hometown']) ? htmlspecialchars($profile['hometown'], ENT_COMPAT,'UTF-8',false) : '');
extend the directory profiles a bit more
2013-12-25 09:57:04 +00:00
get rid of xtag
2012-12-27 22:36:06 +00:00
$clean = array();
more progress on directory import
2012-12-27 02:18:56 +00:00
if(array_key_exists('keywords',$profile) and is_array($profile['keywords'])) {
This was bloody hard to sort out.... but now it's sorted and we can move forward again. Put back xtag for efficient keyword searching, use it in conjunction with xprof['keywords'] to avoid a SQL triple join and group_concat (which simply won't scale on a directory server), figured out how best to work timezone corrected birthdays into the protocol. The directory can even provide an age if one was provided to it. Both of these things need some more work, but we have the mechanisms and a plan how to do it right instead of "how the !@#$ are we going to do pull this off?"
2012-12-28 07:33:37 +00:00
import_directory_keywords($hash,$profile['keywords']);
get rid of xtag
2012-12-27 22:36:06 +00:00
foreach($profile['keywords'] as $kw) {
issue #240 - we were using htmlentities instead of htmlspecialchars in several places, and this was a bit greedy in the set of characters which were converted from utf-8 to HTML entities. Also brought mail attachments up to date so they are rendered identically to item attachments.
2013-12-12 07:13:36 +00:00
$kw = trim(htmlspecialchars($kw,ENT_COMPAT,'UTF-8',false));
small updates
2013-05-09 04:30:54 +00:00
$kw = trim($kw,',');
some fixes to directory keyword searches
2013-09-15 00:20:24 +00:00
$clean[] = $kw;
get rid of xtag
2012-12-27 22:36:06 +00:00
}
more progress on directory import
2012-12-27 02:18:56 +00:00
}
get rid of xtag
2012-12-27 22:36:06 +00:00
$arr['xprof_keywords'] = implode(' ',$clean);
provide a "safe search" backend and allow for self-censorship using nsfw or adult profile keywords. Eventually the directories will be forced to mark adult profiles and sync this knowledge between them. At the moment there's no way to do an unsafe search, but we really just need a checkbox and pass the value through directory to dirsearch on the back end, and some will want this as a pconfig.
2013-09-20 02:50:13 +00:00
// Self censored, make it so
// These are not translated, so the German "erwachsenen" keyword will not censor the directory profile. Only the English form - "adult".
structure for channel unions
2013-09-26 23:25:28 +00:00
provide a "safe search" backend and allow for self-censorship using nsfw or adult profile keywords. Eventually the directories will be forced to mark adult profiles and sync this knowledge between them. At the moment there's no way to do an unsafe search, but we really just need a checkbox and pass the value through directory to dirsearch on the back end, and some will want this as a pconfig.
2013-09-20 02:50:13 +00:00
if(in_arrayi('nsfw',$clean) || in_arrayi('adult',$clean)) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update xchan set xchan_flags = (xchan_flags | %d) where xchan_hash = '%s'",
structure for channel unions
2013-09-26 23:25:28 +00:00
intval(XCHAN_FLAGS_SELFCENSORED),
missing hash
2013-09-20 04:13:45 +00:00
dbesc($hash)
provide a "safe search" backend and allow for self-censorship using nsfw or adult profile keywords. Eventually the directories will be forced to mark adult profiles and sync this knowledge between them. At the moment there's no way to do an unsafe search, but we really just need a checkbox and pass the value through directory to dirsearch on the back end, and some will want this as a pconfig.
2013-09-20 02:50:13 +00:00
);
}
more progress on directory import
2012-12-27 02:18:56 +00:00
$r = q("select * from xprof where xprof_hash = '%s' limit 1",
dbesc($hash)
);
if($r) {
change the way profiles are imported
2013-05-01 01:16:51 +00:00
$update = false;
foreach($r[0] as $k => $v) {
if((array_key_exists($k,$arr)) && ($arr[$k] != $v)) {
sql error in profile_activity
2014-08-31 23:24:05 +00:00
logger('import_directory_profile: update ' . $k . ' => ' . $arr[$k]);
change the way profiles are imported
2013-05-01 01:16:51 +00:00
$update = true;
break;
}
}
if($update) {
$x = q("update xprof set
xprof_desc = '%s',
xprof_dob = '%s',
missing comma
2013-07-03 05:29:24 +00:00
xprof_age = %d,
change the way profiles are imported
2013-05-01 01:16:51 +00:00
xprof_gender = '%s',
xprof_marital = '%s',
xprof_sexual = '%s',
xprof_locale = '%s',
xprof_region = '%s',
xprof_postcode = '%s',
xprof_country = '%s',
extend the directory profiles a bit more
2013-12-25 09:57:04 +00:00
xprof_about = '%s',
xprof_homepage = '%s',
xprof_hometown = '%s',
change the way profiles are imported
2013-05-01 01:16:51 +00:00
xprof_keywords = '%s'
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
where xprof_hash = '%s'",
change the way profiles are imported
2013-05-01 01:16:51 +00:00
dbesc($arr['xprof_desc']),
dbesc($arr['xprof_dob']),
age was not syncing to the directory
2013-07-03 05:22:13 +00:00
intval($arr['xprof_age']),
change the way profiles are imported
2013-05-01 01:16:51 +00:00
dbesc($arr['xprof_gender']),
dbesc($arr['xprof_marital']),
dbesc($arr['xprof_sexual']),
dbesc($arr['xprof_locale']),
dbesc($arr['xprof_region']),
dbesc($arr['xprof_postcode']),
dbesc($arr['xprof_country']),
extend the directory profiles a bit more
2013-12-25 09:57:04 +00:00
dbesc($arr['xprof_about']),
dbesc($arr['xprof_homepage']),
dbesc($arr['xprof_hometown']),
change the way profiles are imported
2013-05-01 01:16:51 +00:00
dbesc($arr['xprof_keywords']),
dbesc($arr['xprof_hash'])
);
}
more progress on directory import
2012-12-27 02:18:56 +00:00
}
else {
change the way profiles are imported
2013-05-01 01:16:51 +00:00
$update = true;
sql error in profile_activity
2014-08-31 23:24:05 +00:00
logger('import_directory_profile: new profile ');
typo in import_directory_profile
2014-01-05 23:51:44 +00:00
$x = q("insert into xprof (xprof_hash, xprof_desc, xprof_dob, xprof_age, xprof_gender, xprof_marital, xprof_sexual, xprof_locale, xprof_region, xprof_postcode, xprof_country, xprof_about, xprof_homepage, xprof_hometown, xprof_keywords) values ('%s', '%s', '%s', %d, '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s') ",
more progress on directory import
2012-12-27 02:18:56 +00:00
dbesc($arr['xprof_hash']),
dbesc($arr['xprof_desc']),
dbesc($arr['xprof_dob']),
age was not syncing to the directory
2013-07-03 05:22:13 +00:00
intval($arr['xprof_age']),
more progress on directory import
2012-12-27 02:18:56 +00:00
dbesc($arr['xprof_gender']),
dbesc($arr['xprof_marital']),
dbesc($arr['xprof_sexual']),
dbesc($arr['xprof_locale']),
dbesc($arr['xprof_region']),
dbesc($arr['xprof_postcode']),
get rid of xtag
2012-12-27 22:36:06 +00:00
dbesc($arr['xprof_country']),
extend the directory profiles a bit more
2013-12-25 09:57:04 +00:00
dbesc($arr['xprof_about']),
dbesc($arr['xprof_homepage']),
dbesc($arr['xprof_hometown']),
get rid of xtag
2012-12-27 22:36:06 +00:00
dbesc($arr['xprof_keywords'])
more progress on directory import
2012-12-27 02:18:56 +00:00
);
}
master/standalone directory profile update
2012-12-27 03:39:37 +00:00
start saving site info
2013-07-24 05:33:56 +00:00
$d = array('xprof' => $arr, 'profile' => $profile, 'update' => $update);
call_hooks('import_directory_profile', $d);
suppress creating the directory update record for profile updates which are part of the normal import_xchan sequence - otherwise we get two for every change. Create it normally if we are called with a profile_update message and don't go through the whole import_xchan thing.
2013-09-26 04:22:36 +00:00
if(($d['update']) && (! $suppress_update))
populate the new fields in the directory updates table
2013-09-25 03:13:53 +00:00
update_modtime($arr['xprof_hash'],random_string() . '@' . get_app()->get_hostname(), $addr, $ud_flags);
start saving site info
2013-07-24 05:33:56 +00:00
return $d['update'];
more progress on directory import
2012-12-27 02:18:56 +00:00
}
This was bloody hard to sort out.... but now it's sorted and we can move forward again. Put back xtag for efficient keyword searching, use it in conjunction with xprof['keywords'] to avoid a SQL triple join and group_concat (which simply won't scale on a directory server), figured out how best to work timezone corrected birthdays into the protocol. The directory can even provide an age if one was provided to it. Both of these things need some more work, but we have the mechanisms and a plan how to do it right instead of "how the !@#$ are we going to do pull this off?"
2012-12-28 07:33:37 +00:00
function import_directory_keywords($hash,$keywords) {
$existing = array();
$r = q("select * from xtag where xtag_hash = '%s'",
dbesc($hash)
);
if($r) {
foreach($r as $rr)
$existing[] = $rr['xtag_term'];
}
$clean = array();
foreach($keywords as $kw) {
issue #240 - we were using htmlentities instead of htmlspecialchars in several places, and this was a bit greedy in the set of characters which were converted from utf-8 to HTML entities. Also brought mail attachments up to date so they are rendered identically to item attachments.
2013-12-12 07:13:36 +00:00
$kw = trim(htmlspecialchars($kw,ENT_COMPAT,'UTF-8',false));
trim commas from keywords
2013-09-19 08:45:06 +00:00
$kw = trim($kw,',');
This was bloody hard to sort out.... but now it's sorted and we can move forward again. Put back xtag for efficient keyword searching, use it in conjunction with xprof['keywords'] to avoid a SQL triple join and group_concat (which simply won't scale on a directory server), figured out how best to work timezone corrected birthdays into the protocol. The directory can even provide an age if one was provided to it. Both of these things need some more work, but we have the mechanisms and a plan how to do it right instead of "how the !@#$ are we going to do pull this off?"
2012-12-28 07:33:37 +00:00
$clean[] = $kw;
}
foreach($existing as $x) {
if(! in_array($x,$clean))
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("delete from xtag where xtag_hash = '%s' and xtag_term = '%s'",
This was bloody hard to sort out.... but now it's sorted and we can move forward again. Put back xtag for efficient keyword searching, use it in conjunction with xprof['keywords'] to avoid a SQL triple join and group_concat (which simply won't scale on a directory server), figured out how best to work timezone corrected birthdays into the protocol. The directory can even provide an age if one was provided to it. Both of these things need some more work, but we have the mechanisms and a plan how to do it right instead of "how the !@#$ are we going to do pull this off?"
2012-12-28 07:33:37 +00:00
dbesc($hash),
dbesc($x)
);
}
foreach($clean as $x) {
if(! in_array($x,$existing))
small updates
2013-05-09 04:30:54 +00:00
$r = q("insert into xtag ( xtag_hash, xtag_term) values ( '%s' ,'%s' )",
This was bloody hard to sort out.... but now it's sorted and we can move forward again. Put back xtag for efficient keyword searching, use it in conjunction with xprof['keywords'] to avoid a SQL triple join and group_concat (which simply won't scale on a directory server), figured out how best to work timezone corrected birthdays into the protocol. The directory can even provide an age if one was provided to it. Both of these things need some more work, but we have the mechanisms and a plan how to do it right instead of "how the !@#$ are we going to do pull this off?"
2012-12-28 07:33:37 +00:00
dbesc($hash),
dbesc($x)
);
}
update modification time whenever a directory related attribute is changed
2013-03-14 00:27:17 +00:00
}
populate the new fields in the directory updates table
2013-09-25 03:13:53 +00:00
function update_modtime($hash,$guid,$addr,$flags = 0) {
don't store update records if we're not a directory server of some kind
2013-11-05 04:43:32 +00:00
$dirmode = intval(get_config('system','directory_mode'));
if($dirmode == DIRECTORY_MODE_NORMAL)
return;
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
if($flags) {
q("insert into updates (ud_hash, ud_guid, ud_date, ud_flags, ud_addr ) values ( '%s', '%s', '%s', %d, '%s' )",
dbesc($hash),
dbesc($guid),
dbesc(datetime_convert()),
intval($flags),
dbesc($addr)
);
}
else {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and not (ud_flags & %d)>0 ",
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
intval(UPDATE_FLAGS_UPDATED),
sql wrong number of args
2013-10-15 04:09:39 +00:00
dbesc($addr),
directory sync - this will either work, or it won't work, or it will possibly recurse and blow up the matrix. Hard to say. Do you feel lucky? Well do ya' ... punk? Rule #1 - don't mess with anything unless it's blowing up the matrix. If it doesn't blow up the matrix, but doesn't work, just let it go and let's figure out what it is doing and what it isn't doing. The flow is as follows: Once a day go out to all the directory servers besides yourself and grab a list of updates. This happens in the poller. If we've never seen them before add them to the updates table. The poller also looks to see if we're a directory server and have updates that haven't yet been processed. It calls onedirsync.php to process each one. If we contact the channel to update and don't find anything (we're just doing a basic zot_finger), set a ud_last timestamp. If this is set we will only try once a day for seven days. Then we stop trying to update. This will probably cause a spike the first time through because you haven't seen any updates before, but we spread out the load over your delivery interval.
2013-10-01 01:33:27 +00:00
intval(UPDATE_FLAGS_UPDATED)
);
}
update modification time whenever a directory related attribute is changed
2013-03-14 00:27:17 +00:00
}
start saving site info
2013-07-24 05:33:56 +00:00
function import_site($arr,$pubkey) {
if( (! is_array($arr)) || (! $arr['url']) || (! $arr['url_sig']))
return false;
if(! rsa_verify($arr['url'],base64url_decode($arr['url_sig']),$pubkey)) {
logger('import_site: bad url_sig');
return false;
}
$update = false;
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
$exists = false;
start saving site info
2013-07-24 05:33:56 +00:00
$r = q("select * from site where site_url = '%s' limit 1",
dbesc($arr['url'])
);
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
if($r) {
$exists = true;
$siterecord = $r[0];
}
start saving site info
2013-07-24 05:33:56 +00:00
$site_directory = 0;
if($arr['directory_mode'] == 'normal')
$site_directory = DIRECTORY_MODE_NORMAL;
if($arr['directory_mode'] == 'primary')
$site_directory = DIRECTORY_MODE_PRIMARY;
if($arr['directory_mode'] == 'secondary')
$site_directory = DIRECTORY_MODE_SECONDARY;
if($arr['directory_mode'] == 'standalone')
$site_directory = DIRECTORY_MODE_STANDALONE;
$register_policy = 0;
if($arr['register_policy'] == 'closed')
$register_policy = REGISTER_CLOSED;
if($arr['register_policy'] == 'open')
$register_policy = REGISTER_OPEN;
if($arr['register_policy'] == 'approve')
$register_policy = REGISTER_APPROVE;
public site list (will take a few days to populate, assuming folks have updated their site access policy which old sites do not have)
2013-09-06 05:00:06 +00:00
$access_policy = 0;
if(array_key_exists('access_policy',$arr)) {
if($arr['access_policy'] === 'private')
$access_policy = ACCESS_PRIVATE;
if($arr['access_policy'] === 'paid')
$access_policy = ACCESS_PAID;
if($arr['access_policy'] === 'free')
$access_policy = ACCESS_FREE;
add "tiered" access policy, add tagcloud to directory pages
2013-09-19 00:47:26 +00:00
if($arr['access_policy'] === 'tiered')
$access_policy = ACCESS_TIERED;
public site list (will take a few days to populate, assuming folks have updated their site access policy which old sites do not have)
2013-09-06 05:00:06 +00:00
}
To be listed as a public site, you need to be an https site with a valid cert. If you don't make the cut, you will either not be listed as a public site or you will be de-listed. Period. This is non-negotiable.
2014-02-07 08:50:04 +00:00
// don't let insecure sites register as public hubs
if(strpos($arr['url'],'https://') === false)
$access_policy = ACCESS_PRIVATE;
if($access_policy != ACCESS_PRIVATE) {
$x = z_fetch_url($arr['url'] . '/siteinfo/json');
if(! $x['success'])
$access_policy = ACCESS_PRIVATE;
}
issue #240 - we were using htmlentities instead of htmlspecialchars in several places, and this was a bit greedy in the set of characters which were converted from utf-8 to HTML entities. Also brought mail attachments up to date so they are rendered identically to item attachments.
2013-12-12 07:13:36 +00:00
$directory_url = htmlspecialchars($arr['directory_url'],ENT_COMPAT,'UTF-8',false);
$url = htmlspecialchars($arr['url'],ENT_COMPAT,'UTF-8',false);
$sellpage = htmlspecialchars($arr['sellpage'],ENT_COMPAT,'UTF-8',false);
$site_location = htmlspecialchars($arr['location'],ENT_COMPAT,'UTF-8',false);
provide backend storage and declaration of directory realm
2014-08-16 23:10:35 +00:00
$site_realm = htmlspecialchars($arr['realm'],ENT_COMPAT,'UTF-8',false);
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
if($exists) {
if(($siterecord['site_flags'] != $site_directory)
|| ($siterecord['site_access'] != $access_policy)
|| ($siterecord['site_directory'] != $directory_url)
site sellpage links
2013-09-18 05:27:51 +00:00
|| ($siterecord['site_sellpage'] != $sellpage)
allow zot public providers to list their location, as a non US-based server could be a strong selling point.
2013-10-30 08:58:45 +00:00
|| ($siterecord['site_location'] != $site_location)
provide backend storage and declaration of directory realm
2014-08-16 23:10:35 +00:00
|| ($siterecord['site_register'] != $register_policy)
|| ($siterecord['site_realm'] != $site_realm)) {
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
$update = true;
found it
2013-09-16 06:02:18 +00:00
// logger('import_site: input: ' . print_r($arr,true));
// logger('import_site: stored: ' . print_r($siterecord,true));
more logging
2013-09-16 05:19:24 +00:00
provide backend storage and declaration of directory realm
2014-08-16 23:10:35 +00:00
$r = q("update site set site_location = '%s', site_flags = %d, site_access = %d, site_directory = '%s', site_register = %d, site_update = '%s', site_sellpage = '%s', site_realm = '%s'
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
where site_url = '%s'",
allow zot public providers to list their location, as a non US-based server could be a strong selling point.
2013-10-30 08:58:45 +00:00
dbesc($site_location),
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
intval($site_directory),
intval($access_policy),
dbesc($directory_url),
intval($register_policy),
dbesc(datetime_convert()),
site sellpage links
2013-09-18 05:27:51 +00:00
dbesc($sellpage),
provide backend storage and declaration of directory realm
2014-08-16 23:10:35 +00:00
dbesc($site_realm),
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
dbesc($url)
);
if(! $r) {
logger('import_site: update failed. ' . print_r($arr,true));
}
start saving site info
2013-07-24 05:33:56 +00:00
}
}
else {
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
$update = true;
some more work on realms
2014-08-18 02:06:56 +00:00
$r = q("insert into site ( site_location, site_url, site_access, site_flags, site_update, site_directory, site_register, site_sellpage, site_realm )
values ( '%s', '%s', %d, %d, '%s', '%s', %d, '%s', '%s' )",
allow zot public providers to list their location, as a non US-based server could be a strong selling point.
2013-10-30 08:58:45 +00:00
dbesc($site_location),
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
dbesc($url),
public site list (will take a few days to populate, assuming folks have updated their site access policy which old sites do not have)
2013-09-06 05:00:06 +00:00
intval($access_policy),
access policy and directory mode were crossed on site table creation
2013-10-30 08:30:37 +00:00
intval($site_directory),
start saving site info
2013-07-24 05:33:56 +00:00
dbesc(datetime_convert()),
don't create a directory sync notification (entry in the updates table) if the site record was updated - without checking first to see if anything changed. This is causing lots of sync entries when nothing changed to warrant it.
2013-09-16 02:04:11 +00:00
dbesc($directory_url),
site sellpage links
2013-09-18 05:27:51 +00:00
intval($register_policy),
some more work on realms
2014-08-18 02:06:56 +00:00
dbesc($sellpage),
dbesc($site_realm)
start saving site info
2013-07-24 05:33:56 +00:00
);
if(! $r) {
logger('import_site: record create failed. ' . print_r($arr,true));
}
}
found it
2013-09-16 06:02:18 +00:00
return $update;
start saving site info
2013-07-24 05:33:56 +00:00
}
more work on clone sync
2013-07-25 23:00:04 +00:00
/**
* Send a zot packet to all hubs where this channel is duplicated, refreshing
* such things as personal settings, channel permissions, address book updates, etc.
*/
the sending side of clone syncing of collection/privacy_group information. The receiving side is not yet implemented.
2014-07-14 09:37:41 +00:00
function build_sync_packet($uid = 0, $packet = null, $groups_changed = false) {
more work on clone sync
2013-07-25 23:00:04 +00:00
$a = get_app();
logger('build_sync_packet');
build_sync_packet was looking at stale channel info - load it fresh from the DB.
2013-07-26 03:10:53 +00:00
more work on clone sync
2013-07-25 23:00:04 +00:00
if(! $uid)
$uid = local_user();
if(! $uid)
return;
Implements a new permission level PERMS_PENDING With this a user can allow some action to any user which connects to them, even before they've connected back. Ref. https://mobiliza.org.br/display/478d9e71eaf55748dc646d3990651d6d34cfb7db5c38360538ec730ca3ccf908@zothub.com Also some code cleanup and an alternative logic for handling notifications of permission changes in zot.php. This assumes that private posts are still restricted to people in your addressbook. Regardless of your global permissions, a pending channel won't get private posts, even if the post only has a deny clause not matching the pending channel.
2014-06-17 07:26:18 +00:00
$r = q("select * from channel where channel_id = %d limit 1",
build_sync_packet was looking at stale channel info - load it fresh from the DB.
2013-07-26 03:10:53 +00:00
intval($uid)
);
if(! $r)
return;
Implements a new permission level PERMS_PENDING With this a user can allow some action to any user which connects to them, even before they've connected back. Ref. https://mobiliza.org.br/display/478d9e71eaf55748dc646d3990651d6d34cfb7db5c38360538ec730ca3ccf908@zothub.com Also some code cleanup and an alternative logic for handling notifications of permission changes in zot.php. This assumes that private posts are still restricted to people in your addressbook. Regardless of your global permissions, a pending channel won't get private posts, even if the post only has a deny clause not matching the pending channel.
2014-06-17 07:26:18 +00:00
build_sync_packet was looking at stale channel info - load it fresh from the DB.
2013-07-26 03:10:53 +00:00
$channel = $r[0];
more work on clone sync
2013-07-25 23:00:04 +00:00
$h = q("select * from hubloc where hubloc_hash = '%s'",
dbesc($channel['channel_hash'])
);
if(! $h)
return;
$synchubs = array();
foreach($h as $x) {
if($x['hubloc_host'] == $a->get_hostname())
continue;
$synchubs[] = $x;
}
if(! $synchubs)
return;
$r = q("select xchan_guid, xchan_guid_sig from xchan where xchan_hash = '%s' limit 1",
dbesc($channel['channel_hash'])
);
if(! $r)
return;
$env_recips = array();
$env_recips[] = array('guid' => $r[0]['xchan_guid'],'guid_sig' => $r[0]['xchan_guid_sig']);
$info = (($packet) ? $packet : array());
$info['type'] = 'channel_sync';
add an encoding type to formatted zot communication structures so that we can later add alternate encodings.
2014-07-24 22:55:01 +00:00
$info['encoding'] = 'red'; // note: not zot, this packet is very red specific
more work on clone sync
2013-07-25 23:00:04 +00:00
if(array_key_exists($uid,$a->config) && array_key_exists('transient',$a->config[$uid])) {
$settings = $a->config[$uid]['transient'];
if($settings) {
$info['config'] = $settings;
}
}
if($channel) {
$info['channel'] = array();
foreach($channel as $k => $v) {
// filter out any joined tables like xchan
if(strpos($k,'channel_') !== 0)
continue;
// don't pass these elements, they should not be synchronised
encrypt private messages on disk - there are still a couple of places where the text is leaked in the logs during processing.
2013-07-29 04:04:03 +00:00
$disallowed = array('channel_id','channel_account_id','channel_primary','channel_prvkey','channel_address');
more work on clone sync
2013-07-25 23:00:04 +00:00
if(in_array($k,$disallowed))
continue;
$info['channel'][$k] = $v;
}
}
the sending side of clone syncing of collection/privacy_group information. The receiving side is not yet implemented.
2014-07-14 09:37:41 +00:00
if($groups_changed) {
limit fields to what we require
2014-07-15 04:52:20 +00:00
$r = q("select hash as collection, visible, deleted, name from groups where uid = %d",
the sending side of clone syncing of collection/privacy_group information. The receiving side is not yet implemented.
2014-07-14 09:37:41 +00:00
intval($uid)
);
if($r)
$info['collections'] = $r;
index collection sync info by hash rather than id
2014-07-15 04:49:48 +00:00
$r = q("select groups.hash as collection, group_member.xchan as member from groups left join group_member on groups.id = group_member.gid where group_member.uid = %d",
the sending side of clone syncing of collection/privacy_group information. The receiving side is not yet implemented.
2014-07-14 09:37:41 +00:00
intval($uid)
);
if($r)
$info['collection_members'] = $r;
}
more work on clone sync
2013-07-25 23:00:04 +00:00
$interval = ((get_config('system','delivery_interval') !== false)
? intval(get_config('system','delivery_interval')) : 2 );
log the sync packets locally at LOGGER_DATA level
2013-07-26 02:50:55 +00:00
logger('build_sync_packet: packet: ' . print_r($info,true), LOGGER_DATA);
more work on clone sync
2013-07-25 23:00:04 +00:00
foreach($synchubs as $hub) {
$hash = random_string();
$n = zot_build_packet($channel,'notify',$env_recips,$hub['hubloc_sitekey'],$hash);
this should improve queue performance dramatically.
2014-03-25 02:40:36 +00:00
q("insert into outq ( outq_hash, outq_account, outq_channel, outq_driver, outq_posturl, outq_async, outq_created, outq_updated, outq_notify, outq_msg ) values ( '%s', %d, %d, '%s', '%s', %d, '%s', '%s', '%s', '%s' )",
more work on clone sync
2013-07-25 23:00:04 +00:00
dbesc($hash),
intval($channel['channel_account']),
intval($channel['channel_id']),
this should improve queue performance dramatically.
2014-03-25 02:40:36 +00:00
dbesc('zot'),
more work on clone sync
2013-07-25 23:00:04 +00:00
dbesc($hub['hubloc_callback']),
intval(1),
dbesc(datetime_convert()),
dbesc(datetime_convert()),
dbesc($n),
dbesc(json_encode($info))
);
proc_run('php','include/deliver.php',$hash);
if($interval)
@time_sleep_until(microtime(true) + (float) $interval);
}
}
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
function process_channel_sync_delivery($sender,$arr,$deliveries) {
add gz to access denied files to block rotated logs from public access
2013-07-26 10:50:46 +00:00
// FIXME - this will sync red structures (channel, pconfig and abook). Eventually we need to make this application agnostic.
honour service class restrictions for total_identities, total_channels ("friends") and total_feeds both when importing channels and subsequently when syncing clones. Limits are based on the local system - additional entries are silently dropped.
2014-09-16 00:17:00 +00:00
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
$result = array();
foreach($deliveries as $d) {
$r = q("select * from channel where channel_hash = '%s' limit 1",
dbesc($d['hash'])
);
if(! $r) {
$result[] = array($d['hash'],'not found');
continue;
}
$channel = $r[0];
php warnings
2014-11-03 03:22:18 +00:00
$max_friends = service_class_fetch($channel['channel_id'],'total_channels');
$max_feeds = account_service_class_fetch($channel['channel_account_id'],'total_feeds');
honour service class restrictions for total_identities, total_channels ("friends") and total_feeds both when importing channels and subsequently when syncing clones. Limits are based on the local system - additional entries are silently dropped.
2014-09-16 00:17:00 +00:00
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
if($channel['channel_hash'] != $sender['hash']) {
logger('process_channel_sync_delivery: possible forgery. Sender ' . $sender['hash'] . ' is not ' . $channel['channel_hash']);
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'channel mismatch',$channel['channel_name'],'');
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
continue;
}
if(array_key_exists('config',$arr) && is_array($arr['config']) && count($arr['config'])) {
foreach($arr['config'] as $cat => $k) {
foreach($arr['config'][$cat] as $k => $v)
set_pconfig($channel['channel_id'],$cat,$k,$v);
}
}
if(array_key_exists('channel',$arr) && is_array($arr['channel']) && count($arr['channel'])) {
Check for the correct DB name before pushing.
2014-02-11 03:20:55 +00:00
$disallowed = array('channel_id','channel_account_id','channel_primary','channel_prvkey', 'channel_address', 'channel_notifyflags');
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
$clean = array();
foreach($arr['channel'] as $k => $v) {
if(in_array($k,$disallowed))
continue;
$clean[$k] = $v;
}
if(count($clean)) {
foreach($clean as $k => $v) {
$r = dbq("UPDATE channel set " . dbesc($k) . " = '" . dbesc($v)
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
. "' where channel_id = " . intval($channel['channel_id']) );
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
}
}
}
collection sync
2014-07-17 04:18:28 +00:00
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
if(array_key_exists('abook',$arr) && is_array($arr['abook']) && count($arr['abook'])) {
honour service class restrictions for total_identities, total_channels ("friends") and total_feeds both when importing channels and subsequently when syncing clones. Limits are based on the local system - additional entries are silently dropped.
2014-09-16 00:17:00 +00:00
$total_friends = 0;
$total_feeds = 0;
$r = q("select abook_id, abook_flags from abook where abook_channel = %d",
intval($channel['channel_id'])
);
if($r) {
// don't count yourself
found the sucker
2014-09-22 23:23:59 +00:00
$total_friends = ((count($r) > 0) ? count($r) - 1 : 0);
honour service class restrictions for total_identities, total_channels ("friends") and total_feeds both when importing channels and subsequently when syncing clones. Limits are based on the local system - additional entries are silently dropped.
2014-09-16 00:17:00 +00:00
foreach($r as $rr)
if($rr['abook_flags'] & ABOOK_FLAG_FEED)
$total_feeds ++;
}
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
$disallowed = array('abook_id','abook_account','abook_channel');
foreach($arr['abook'] as $abook) {
perform remote discovery on abook sync of nomadic clones if remote channel (xchan) is unknown
2014-02-10 03:45:22 +00:00
found the sucker
2014-09-22 23:23:59 +00:00
$clean = array();
clone sync of "unfriend" actions
2014-08-06 00:47:17 +00:00
if($abook['abook_xchan'] && $abook['entry_deleted']) {
logger('process_channel_sync_delivery: removing abook entry for ' . $abook['abook_xchan']);
require_once('include/Contact.php');
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select abook_id, abook_flags from abook where abook_xchan = '%s' and abook_channel = %d and not ( abook_flags & %d )>0 limit 1",
clone sync of "unfriend" actions
2014-08-06 00:47:17 +00:00
dbesc($abook['abook_xchan']),
intval($channel['channel_id']),
intval(ABOOK_FLAG_SELF)
);
honour service class restrictions for total_identities, total_channels ("friends") and total_feeds both when importing channels and subsequently when syncing clones. Limits are based on the local system - additional entries are silently dropped.
2014-09-16 00:17:00 +00:00
if($r) {
clone sync of "unfriend" actions
2014-08-06 00:47:17 +00:00
contact_remove($channel['channel_id'],$r[0]['abook_id']);
honour service class restrictions for total_identities, total_channels ("friends") and total_feeds both when importing channels and subsequently when syncing clones. Limits are based on the local system - additional entries are silently dropped.
2014-09-16 00:17:00 +00:00
if($total_friends)
$total_friends --;
if($r[0]['abook_flags'] & ABOOK_FLAG_FEED)
$total_feeds --;
}
clone sync of "unfriend" actions
2014-08-06 00:47:17 +00:00
continue;
}
perform remote discovery on abook sync of nomadic clones if remote channel (xchan) is unknown
2014-02-10 03:45:22 +00:00
// Perform discovery if the referenced xchan hasn't ever been seen on this hub.
// This relies on the undocumented behaviour that red sites send xchan info with the abook
if($abook['abook_xchan'] && $abook['xchan_address']) {
$h = zot_get_hublocs($abook['abook_xchan']);
if(! $h) {
$f = zot_finger($abook['xchan_address'],$channel);
if(! $f['success']) {
logger('process_channel_sync_delivery: abook not probe-able' . $abook['xchan_address']);
continue;
}
$j = json_decode($f['body'],true);
php warnings
2014-11-03 03:22:18 +00:00
if(! ($j['success'] && $j['guid'])) {
perform remote discovery on abook sync of nomadic clones if remote channel (xchan) is unknown
2014-02-10 03:45:22 +00:00
logger('process_channel_sync_delivery: probe failed.');
continue;
}
$x = import_xchan($j);
if(! $x['success']) {
logger('process_channel_sync_delivery: import failed.');
continue;
}
}
}
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
foreach($abook as $k => $v) {
clone sync was trying to update xchan info in the abook loop.
2013-08-27 03:58:07 +00:00
if(in_array($k,$disallowed) || (strpos($k,'abook') !== 0))
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
continue;
$clean[$k] = $v;
}
if(! array_key_exists('abook_xchan',$clean))
continue;
make sure new contacts who are not yet in the address book are sync'd across clone instances
2013-09-09 03:38:15 +00:00
$r = q("select * from abook where abook_xchan = '%s' and abook_channel = %d limit 1",
dbesc($clean['abook_xchan']),
intval($channel['channel_id'])
);
// make sure we have an abook entry for this xchan on this system
if(! $r) {
honour service class restrictions for total_identities, total_channels ("friends") and total_feeds both when importing channels and subsequently when syncing clones. Limits are based on the local system - additional entries are silently dropped.
2014-09-16 00:17:00 +00:00
if($max_friends !== false && $total_friends > $max_friends) {
logger('process_channel_sync_delivery: total_channels service class limit exceeded');
continue;
}
if($max_feeds !== false && ($clean['abook_flags'] & ABOOK_FLAG_FEED) && $total_feeds > $max_feeds) {
logger('process_channel_sync_delivery: total_feeds service class limit exceeded');
continue;
}
make sure new contacts who are not yet in the address book are sync'd across clone instances
2013-09-09 03:38:15 +00:00
q("insert into abook ( abook_xchan, abook_channel ) values ('%s', %d ) ",
dbesc($clean['abook_xchan']),
intval($channel['channel_id'])
);
honour service class restrictions for total_identities, total_channels ("friends") and total_feeds both when importing channels and subsequently when syncing clones. Limits are based on the local system - additional entries are silently dropped.
2014-09-16 00:17:00 +00:00
$total_friends ++;
if($clean['abook_flags'] & ABOOK_FLAG_FEED)
$total_feeds ++;
make sure new contacts who are not yet in the address book are sync'd across clone instances
2013-09-09 03:38:15 +00:00
}
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
if(count($clean)) {
foreach($clean as $k => $v) {
$r = dbq("UPDATE abook set " . dbesc($k) . " = '" . dbesc($v)
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
. "' where abook_xchan = '" . dbesc($clean['abook_xchan']) . "' and abook_channel = " . intval($channel['channel_id']));
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
}
}
}
}
sync profiles to nomadic clones
2014-06-18 00:34:51 +00:00
collection sync
2014-07-17 04:18:28 +00:00
// sync collections (privacy groups) oh joy...
if(array_key_exists('collections',$arr) && is_array($arr['collections']) && count($arr['collections'])) {
$x = q("select * from groups where uid = %d",
intval($channel['channel_id'])
);
foreach($arr['collections'] as $cl) {
$found = false;
if($x) {
foreach($x as $y) {
if($cl['collection'] == $y['hash']) {
$found = true;
break;
}
}
if($found) {
if(($y['name'] != $cl['name'])
|| ($y['visible'] != $cl['visible'])
|| ($y['deleted'] != $cl['deleted'])) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update groups set name = '%s', visible = %d, deleted = %d where hash = '%s' and uid = %d",
collection sync
2014-07-17 04:18:28 +00:00
dbesc($cl['name']),
intval($cl['visible']),
intval($cl['deleted']),
dbesc($cl['hash']),
intval($channel['channel_id'])
);
}
if(intval($cl['deleted']) && (! intval($y['deleted']))) {
q("delete from group_member where gid = %d",
intval($y['id'])
);
}
}
}
if(! $found) {
$r = q("INSERT INTO `groups` ( hash, uid, visible, deleted, name )
VALUES( '%s', %d, %d, %d, '%s' ) ",
dbesc($cl['collection']),
intval($channel['channel_id']),
intval($cl['visible']),
intval($cl['deleted']),
php warnings
2014-11-03 03:22:18 +00:00
dbesc($cl['name'])
);
collection sync
2014-07-17 04:18:28 +00:00
}
collection sync issue
2014-07-17 22:53:07 +00:00
// now look for any collections locally which weren't in the list we just received.
// They need to be removed by marking deleted and removing the members.
// This shouldn't happen except for clones created before this function was written.
if($x) {
$found_local = false;
foreach($x as $y) {
foreach($arr['collections'] as $cl) {
if($cl['collection'] == $y['hash']) {
$found_local = true;
break;
}
}
if(! $found_local) {
q("delete from group_member where gid = %d",
intval($y['id'])
);
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("update groups set deleted = 1 where id = %d and uid = %d",
collection sync issue
2014-07-17 22:53:07 +00:00
intval($y['id']),
intval($channel['channel_id'])
);
}
}
}
collection sync
2014-07-17 04:18:28 +00:00
}
// reload the group list with any updates
$x = q("select * from groups where uid = %d",
intval($channel['channel_id'])
);
// now sync the members
if(array_key_exists('collection_members',$arr) &&
is_array($arr['collection_members']) && count($arr['collection_members'])) {
// first sort into groups keyed by the group hash
$members = array();
foreach($arr['collection_members'] as $cm) {
if(! array_key_exists($cm['collection'],$members))
$members[$cm['collection']] = array();
$members[$cm['collection']][] = $cm['member'];
}
// our group list is already synchronised
if($x) {
foreach($x as $y) {
// for each group, loop on members list we just received
foreach($members[$y['hash']] as $member) {
$found = false;
$z = q("select xchan from group_member where gid = %d and uid = %d and xchan = '%s' limit 1",
intval($y['id']),
intval($channel['channel_id']),
dbesc($member)
);
if($z)
$found = true;
// if somebody is in the group that wasn't before - add them
if(! $found) {
q("INSERT INTO `group_member` (`uid`, `gid`, `xchan`)
VALUES( %d, %d, '%s' ) ",
intval($channel['channel_id']),
intval($y['id']),
dbesc($member)
);
}
}
// now retrieve a list of members we have on this site
$m = q("select xchan from group_member where gid = %d and uid = %d",
intval($y['id']),
intval($channel['channel_id'])
);
if($m) {
foreach($m as $mm) {
// if the local existing member isn't in the list we just received - remove them
if(! in_array($mm['xchan'],$members[$y['hash']])) {
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
q("delete from group_member where xchan = '%s' and gid = %d and uid = %d",
collection sync
2014-07-17 04:18:28 +00:00
dbesc($mm['xchan']),
intval($y['id']),
intval($channel['channel_id'])
);
}
}
}
}
}
}
}
sync profiles to nomadic clones
2014-06-18 00:34:51 +00:00
if(array_key_exists('profile',$arr) && is_array($arr['profile']) && count($arr['profile'])) {
$disallowed = array('id','aid','uid');
foreach($arr['profile'] as $profile) {
$x = q("select * from profile where profile_guid = '%s' and uid = %d limit 1",
dbesc($profile['profile_guid']),
intval($channel['channel_id'])
);
if(! $x) {
q("insert into profile ( profile_guid, aid, uid ) values ('%s', %d, %d)",
dbesc($profile['profile_guid']),
intval($channel['channel_account_id']),
intval($channel['channel_id'])
);
$x = q("select * from profile where profile_guid = '%s' and uid = %d limit 1",
dbesc($profile['profile_guid']),
intval($channel['channel_id'])
);
if(! $x)
continue;
}
$clean = array();
foreach($profile as $k => $v) {
if(in_array($k,$disallowed))
continue;
$clean[$k] = $v;
// TODO - check if these are allowed, otherwise we'll error
note work still remaining
2014-06-18 00:36:41 +00:00
// We also need to import local photos if a custom photo is selected
sync profiles to nomadic clones
2014-06-18 00:34:51 +00:00
}
if(count($clean)) {
foreach($clean as $k => $v) {
$r = dbq("UPDATE profile set " . dbesc($k) . " = '" . dbesc($v)
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
. "' where profile_guid = '" . dbesc($profile['profile_guid']) . "' and uid = " . intval($channel['channel_id']));
sync profiles to nomadic clones
2014-06-18 00:34:51 +00:00
}
}
}
}
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
add the message-id to delivery reports. This _could_ allow an enterprising developer to collect the delivery reports for a given post and store them in a DB table so that the sender could track and verify where a message had been sent on a web page and verify the success or failure of those attempts without requiring admin access. (To be fair this would also need an extra flag which hasn't yet been implemented to indicate that a channel created a second delivery chain of the message).
2014-04-11 05:17:35 +00:00
$result[] = array($d['hash'],'channel sync updated',$channel['channel_name'],'');
the rest of the heavy lifting on clone sync - now we're down to some log messages and a whole lot of testing
2013-07-26 05:57:06 +00:00
}
return $result;
}
rpost bbcode tag
2013-10-30 01:46:51 +00:00
// We probably should make rpost discoverable.
function get_rpost_path($observer) {
if(! $observer)
return '';
$parsed = parse_url($observer['xchan_url']);
return $parsed['scheme'] . '://' . $parsed['host'] . (($parsed['port']) ? ':' . $parsed['port'] : '') . '/rpost?f=';
send rpost requests from remote sites back to yours.
2013-10-30 04:49:10 +00:00
}
include re-organisation and more doco, post_to_red fix ampersands in categories
2013-12-04 08:19:29 +00:00
function import_author_zot($x) {
clean up more code duplication
2014-07-15 04:21:24 +00:00
$hash = make_xchan_hash($x['guid'],$x['guid_sig']);
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
$r = q("select hubloc_url from hubloc where hubloc_guid = '%s' and hubloc_guid_sig = '%s' and (hubloc_flags & %d)>0 limit 1",
include re-organisation and more doco, post_to_red fix ampersands in categories
2013-12-04 08:19:29 +00:00
dbesc($x['guid']),
dbesc($x['guid_sig']),
intval(HUBLOC_FLAGS_PRIMARY)
);
if($r) {
logger('import_author_zot: in cache', LOGGER_DEBUG);
return $hash;
}
logger('import_author_zot: entry not in cache - probing: ' . print_r($x,true), LOGGER_DEBUG);
$them = array('hubloc_url' => $x['url'],'xchan_guid' => $x['guid'], 'xchan_guid_sig' => $x['guid_sig']);
if(zot_refresh($them))
return $hash;
return false;
}
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
/**
* @function zot_process_message_request($data)
* If a site receives a comment to a post but finds they have no parent to attach it with, they
* may send a 'request' packet containing the message_id of the missing parent. This is the handler
* for that packet. We will create a message_list array of the entire conversation starting with
* the missing parent and invoke delivery to the sender of the packet.
*
* include/deliver.php (for local delivery) and mod/post.php (for web delivery) detect the existence of
* this 'message_list' at the destination and split it into individual messages which are
* processed/delivered in order.
*
* Called from mod/post.php
*/
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
function zot_process_message_request($data) {
$ret = array('success' => false);
disable request message response until we work through this
2014-11-01 07:36:21 +00:00
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
if(! $data['message_id']) {
$ret['message'] = 'no message_id';
logger('no message_id');
return $ret;
}
$sender = $data['sender'];
$sender_hash = make_xchan_hash($sender['guid'],$sender['guid_sig']);
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
/*
* Find the local channel in charge of this post (the first and only recipient of the request packet)
*/
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
$arr = $data['recipients'][0];
$recip_hash = make_xchan_hash($arr['guid'],$arr['guid_sig']);
$c = q("select * from channel left join xchan on channel_hash = xchan_hash where channel_hash = '%s' limit 1",
dbesc($recip_hash)
);
if(! $c) {
logger('recipient channel not found.');
$ret['message'] .= 'recipient not found.' . EOL;
return $ret;
}
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
/*
* fetch the requested conversation
*/
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
$messages = zot_feed($c[0]['channel_id'],$sender_hash,array('message_id' => $data['message_id']));
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
if($messages) {
$env_recips = null;
$r = q("select hubloc_guid, hubloc_url, hubloc_sitekey, hubloc_network, hubloc_flags, hubloc_callback, hubloc_host
Merge remote-tracking branch 'upstream/master' Conflicts: boot.php include/dba/dba_driver.php include/diaspora.php include/follow.php include/session.php include/zot.php mod/photos.php mod/ping.php
2014-11-13 21:06:31 +00:00
from hubloc where hubloc_hash = '%s' and not (hubloc_flags & %d)>0
PostgreSQL support initial commit There were 11 main types of changes: - UPDATE's and DELETE's sometimes had LIMIT 1 at the end of them. This is not only non-compliant but it would certainly not do what whoever wrote it thought it would. It is likely this mistake was just copied from Friendica. All of these instances, the LIMIT 1 was simply removed. - Bitwise operations (and even some non-zero int checks) erroneously rely on MySQL implicit integer-boolean conversion in the WHERE clauses. This is non-compliant (and bad programming practice to boot). Proper explicit boolean conversions were added. New queries should use proper conventions. - MySQL has a different operator for bitwise XOR than postgres. Rather than add yet another dba_ func, I converted them to "& ~" ("AND NOT") when turning off, and "|" ("OR") when turning on. There were no true toggles (XOR). New queries should refrain from using XOR when not necessary. - There are several fields which the schema has marked as NOT NULL, but the inserts don't specify them. The reason this works is because mysql totally ignores the constraint and adds an empty text default automatically. Again, non-compliant, obviously. In these cases a default of empty text was added. - Several statements rely on a non-standard MySQL feature (http://dev.mysql.com/doc/refman/5.5/en/group-by-handling.html). These queries can all be rewritten to be standards compliant. Interestingly enough, the newly rewritten standards compliant queries run a zillion times faster, even on MySQL. - A couple of function/operator name translations were needed (RAND/RANDOM, GROUP_CONCAT/STRING_AGG, UTC_NOW, REGEXP/~, ^/#) -- assist functions added in the dba_ - INTERVALs: postgres requires quotes around the value, mysql requires that there are not quotes around the value -- assist functions added in the dba_ - NULL_DATE's -- Postgres does not allow the invalid date '0000-00-00 00:00:00' (there is no such thing as year 0 or month 0 or day 0). We use '0001-01-01 00:00:00' for postgres. Conversions are handled in Zot/item packets automagically by quoting all dates with dbescdate(). - char(##) specifications in the schema creates fields with blank spaces that aren't trimmed in the code. MySQL apparently treats char(##) as varchar(##), again, non-compliant. Since postgres works better with text fields anyway, this ball of bugs was simply side-stepped by using 'text' datatype for all text fields in the postgres schema. varchar was used in a couple of places where it actually seemed appropriate (size constraint), but without rigorously vetting that all of the PHP code actually validates data, new bugs might come out from under the rug. - postgres doesn't store nul bytes and a few other non-printables in text fields, even when quoted. bytea fields were used when storing binary data (photo.data, attach.data). A new dbescbin() function was added to handle this transparently. - postgres does not support LIMIT #,# syntax. All databases support LIMIT # OFFSET # syntax. Statements were updated to be standard. These changes require corresponding changes in the coding standards. Please review those before adding any code going forward. Still on my TODO list: - remove quotes from non-reserved identifiers and make reserved identifiers use dba func for quoting - Rewrite search queries for better results (both MySQL and Postgres)
2014-11-13 20:21:58 +00:00
and not (hubloc_status & %d)>0 group by hubloc_sitekey",
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
dbesc($sender_hash),
php warnings
2014-11-03 03:22:18 +00:00
intval(HUBLOC_FLAGS_DELETED),
intval(HUBLOC_OFFLINE)
);
if(! $r) {
logger('no hubs');
return $ret;
}
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
$hubs = $r;
$hublist = array();
$keys = array();
$private = ((array_key_exists('flags',$messages[0]) && in_array('private',$messages[0]['flags'])) ? true : false);
if($private)
$env_recips = array('guid' => $sender['guid'],'guid_sig' => $sender['guid_sig'],'hash' => $sender_hash);
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
$data_packet = json_encode(array('message_list' => $messages));
foreach($hubs as $hub) {
$hash = random_string();
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
/*
* create a notify packet and drop the actual message packet in the queue for pickup
*/
flag notify packets that are sent in response to a message_id request and prevent them from recursing
2014-10-31 11:59:30 +00:00
$n = zot_build_packet($c[0],'notify',$env_recips,(($private) ? $hub['hubloc_sitekey'] : null),$hash,array('message_id' => $data['message_id']));
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
q("insert into outq ( outq_hash, outq_account, outq_channel, outq_driver, outq_posturl, outq_async,
outq_created, outq_updated, outq_notify, outq_msg )
values ( '%s', %d, %d, '%s', '%s', %d, '%s', '%s', '%s', '%s' )",
dbesc($hash),
intval($c[0]['channel_account_id']),
intval($c[0]['channel_id']),
dbesc('zot'),
dbesc($hub['hubloc_callback']),
intval(1),
dbesc(datetime_convert()),
dbesc(datetime_convert()),
dbesc($n),
dbesc($data_packet)
);
I'm going to try again with the conversation request packets. I've looked this over and tested a lot of edge cases, and thought about from every angle I can think of to prevent looping. I don't *think* this can loop. I also doubt that this is the problem at friendicared.de, but I don't know for sure what that problem might be.
2014-11-02 23:28:20 +00:00
/*
* invoke delivery to send out the notify packet
*/
What this checkin does is catch the case where a comment arrived and there's a missing top-level post to match it with. So we'll send a request back to the sender that you've never seen this thread and please send a fresh copy of the entire conversation to date. We could soon have posts in the matrix from different platforms from days gone by, which have been migrated into the modern world. We'll be polite and not deliver these to everybody. However, if someone comments on one of these antique threads we wouldn't be able to see it in our own matrix because we won't have a copy of the parent post. So this rectifies that situation. Be aware that item deletion may need to change to keep "hard deleted" items indefinitely so that they don't keep coming back. We'll have to null out the important data of the former item to accomplish the deletion aspect.
2014-10-31 02:15:03 +00:00
proc_run('php','include/deliver.php',$hash);
}
}
$ret['success'] = true;
return $ret;
}
Reference in a new issue Copy permalink